Chapter 5. Serve and inference language models with Podman on NVIDIA DGX Spark

Procedure

1. Open a terminal on your DGX Spark system, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

2. Pull the Red Hat AI Inference CUDA container image by running the following command:

   $ podman pull registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9:3.5.0-ea.2

   Note

   DGX Spark uses the same vllm-cuda-rhel9 container image as data center NVIDIA GPUs. The container registry automatically provides the AArch64 image variant for the DGX Spark ARM architecture.

3. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

   Note

   DGX OS uses AppArmor instead of SELinux. Skip this step if you are running DGX OS.

4. Create a volume and mount it into the container. Adjust the container permissions so that the container can use it.

   $ mkdir -p rhaii-cache

   $ chmod g+rwX rhaii-cache

5. Append your HF_TOKEN Hugging Face token to the private.env file. Source the private.env file.

   In the following command, replace <your_HF_token> with your Hugging Face access token:

   $ echo "export HF_TOKEN=<your_HF_token>" > private.env

   $ source private.env

6. Verify that the Red Hat AI Inference container can access the NVIDIA GPU on the host by running the following command:

   $ podman run --rm -it \
   --security-opt=label=disable \
   --device nvidia.com/gpu=all \
   registry.redhat.io/{rhaii-registry-namespace}/vllm-cuda-rhel9:{rhaiis-version} \
   nvidia-smi

   The output lists the NVIDIA GB10 GPU. DGX Spark uses unified memory shared between the CPU and GPU, so the Memory-Usage fields show N/A because the GPU does not have dedicated memory that can be reported separately. This is expected and confirms that the container can access the GPU.

7. Start the AI Inference container image.

   $ podman run --rm -it \
   --device nvidia.com/gpu=all \
   --security-opt=label=disable \
   --shm-size=4g -p 8000:8000 \
   --userns=keep-id:uid=1001 \
   --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
   --env "HF_HUB_OFFLINE=0" \
   -v ./rhaii-cache:/opt/app-root/src/.cache:Z \
   registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9:3.5.0-ea.2 \
   --model RedHatAI/Llama-3.2-1B-Instruct-FP8 \
   --tensor-parallel-size 1 \
   --gpu-memory-utilization 0.85 \
   --max-num-seqs 4 \
   --load-format fastsafetensors

   Where:

   --security-opt=label=disable

   Disables SELinux label relabeling for volume mounts. Required for systems where SELinux is enabled. Without this option, the container might fail to start.

   --shm-size=4g -p 8000:8000

   Specifies the shared memory size and port mapping. Increase --shm-size to 8GB if you experience shared memory issues.

   --userns=keep-id:uid=1001

   Maps the host UID to the effective UID of the vLLM process in the container. Alternatively, you can pass --user=0, but this is less secure because it runs vLLM as root inside the container.

   --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN"

   Specifies the Hugging Face API access token. Set and export HF_TOKEN with your Hugging Face token.

   -v ./rhaii-cache:/opt/app-root/src/.cache:Z

   Mounts the cache directory with SELinux context. The :Z suffix is required for systems where SELinux is enabled. On DGX OS, Ubuntu, Debian, or Docker without SELinux, omit the :Z suffix.

   --tensor-parallel-size 1

   Specifies the number of GPUs to use for tensor parallelism. DGX Spark has a single integrated GPU, so this value must be 1.

   --gpu-memory-utilization 0.85

   Specifies the fraction of GPU memory that vLLM can use for model weights, activations, and KV cache. On DGX Spark, the 128 GB memory pool is shared between the CPU, GPU, operating system, and container runtime. Setting this value below the default of 0.9 prevents memory contention that can cause out-of-memory crashes. For more guidance on adjusting this value, see Tuning vLLM for NVIDIA DGX Spark.

   --max-num-seqs 4

   Limits the maximum number of sequences processed concurrently. A lower value reduces memory pressure on the single-GPU workstation and improves per-request latency.

   --load-format fastsafetensors

   Uses the fastsafetensors weight loader instead of the default mmap-based safetensors loader. The default loader can hang during weight loading on DGX Spark due to the unified memory architecture and filesystem behavior. The fastsafetensors loader completes weight loading in approximately 1 second.

   Note

   If you are running RHEL on DGX Spark, add -e LD_PRELOAD= to the podman run command to clear the jemalloc preload. The container image preloads a jemalloc build compiled for 4K memory pages, but RHEL on DGX Spark defaults to 64K memory pages. Without this override, the container stops immediately on startup. This setting does not affect vLLM serving performance. DGX OS and Ubuntu use 4K pages and are not affected.

8. In a separate tab in your terminal, send a test request to the model server

   $ curl -X POST -H "Content-Type: application/json" -d '{
       "prompt": "What is the capital of France?",
       "max_tokens": 50
   }' http://localhost:8000/v1/completions | jq

   Example output

   {
       "id": "cmpl-b84aeda1d5a4485c9cb9ed4a13072fca",
       "object": "text_completion",
       "created": 1746555421,
       "model": "RedHatAI/Llama-3.2-1B-Instruct-FP8",
       "choices": [
           {
               "index": 0,
               "text": " Paris.\nThe capital of France is Paris.",
               "logprobs": null,
               "finish_reason": "stop",
               "stop_reason": null,
               "prompt_logprobs": null
           }
       ],
       "usage": {
           "prompt_tokens": 8,
           "total_tokens": 18,
           "completion_tokens": 10,
           "prompt_tokens_details": null
       }
   }