SupportConsoleDevelopersStart a trialContact

Chapter 10. Distributed Inference with llm-d deployment patterns

Distributed Inference with llm-d supports several deployment patterns that range from a single model on one GPU to disaggregated prefill/decode across many nodes.

You can deploy large language models by using validated topology patterns that balance resource allocation, latency, and throughput based on your model size and hardware constraints. Understanding these patterns helps you select the appropriate deployment strategy for your inference workloads.

Single-node topology

The single-node topology deploys all model components on a single GPU or set of GPUs within one Kubernetes pod. This pattern is ideal for development, testing, and production deployments of smaller models that fit within the memory and compute capacity of available accelerators.

Use single-node topology when:

  • Your model fits within the memory of the available GPUs on a single node
  • You are developing or testing model deployments before scaling
  • Your inference workload requires simple resource management
  • You need the fastest deployment path with minimal configuration

Single-node deployments support hardware profile selection, allowing you to specify the exact accelerator resources required for your model.

Multi-node data-parallel topology

The multi-node data-parallel topology distributes inference requests across many worker nodes, each running a complete copy of the model. This pattern provides horizontal scaling for high-throughput workloads by processing many requests in parallel.

Use multi-node data-parallel topology when:

  • You need to handle high request volumes with parallel processing
  • Your model fits on a single GPU but you require higher throughput than one instance can give
  • You want to distribute inference load across many workers for reliability
  • Your workload benefits from request-level parallelism

This topology requires administrator-created configuration templates to define the worker pool and resource allocation.

Single-node disaggregated prefill/decode topology

The single-node disaggregated prefill/decode topology separates the prefill phase, which is processing input prompts, from the decode phase, which is generating output tokens, on different GPUs within a single node. This pattern optimizes latency by specializing compute resources for each phase of LLM inference.

Use single-node disaggregated prefill/decode topology when:

  • You have at least 2 GPUs and RDMA-capable networking available on a single node
  • Your model has 30 billion or more parameters
  • Your workload is suitable for disaggregated inference

The scheduler component coordinates request routing between prefill and decode pods, enabling KV cache-aware scheduling.

Multi-node disaggregated prefill/decode topology

The multi-node prefill/decode disaggregate topology distributes prefill and decode workloads across many nodes, combining disaggregation with horizontal scaling. This pattern supports the largest models and highest throughput requirements by separating phases and parallelizing within each phase. Use multi-node prefill/decode disaggregate topology when:

  • You need to deploy models that require more GPUs than a single node provides
  • You have RDMA-capable networking such as InfiniBand or RoCE v2 across nodes
  • Your workload is suitable for disaggregated inference.
Wide expert parallelism (WideEP) topology

The WideEP topology distributes Mixture of Experts (MoE) model experts across multiple GPU nodes by using all-to-all communication backends such as DeepEP. This pattern enables cost-effective inference for very large MoE models such as DeepSeek-R1 by splitting expert layers across nodes rather than replicating the full model.

Important

Wide expert parallelism (WideEP) is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Use WideEP topology when:

  • Your model uses a Mixture of Experts architecture
  • The model size exceeds what multi-node data-parallel or prefill/decode disaggregate topologies can handle efficiently
  • You have RDMA-capable networking such as RoCE or InfiniBand for expert parallelism communication
  • You need all GPUs on each node allocated to a single pod for NVLink intranode communication

WideEP deployments use a one-pod-per-node topology where each pod requests all GPUs on the node. The LeaderWorkerSet Operator manages multi-node coordination across pods.

The simplest deployment runs one model on a single GPU node behind an Inference Gateway. The Endpoint Picker (EPP) scheduler is not required for this deployment. This pattern suits development workloads and low-traffic production services.

The following LLMInferenceService custom resource (CR) deploys a single model replica on one GPU behind an Inference Gateway with the Endpoint Picker (EPP) scheduler enabled. This is the simplest deployment pattern for Distributed Inference with llm-d and suits development workloads and low-traffic production services.

The CR configures the following components:

  • A single replica serving the Qwen/Qwen3-0.6B model from Hugging Face.
  • The EPP scheduler for intelligent request routing.
  • Resource requests and limits for the model server container, including one NVIDIA GPU.
  • An HTTPS liveness probe on port 8000 to monitor model server health.
Note

Replace the spec.model.uri and spec.model.name values with your model. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

Single-node LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: single-gpu
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 120
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

  • spec.model.uri: Specifies the model location. The hf:// prefix indicates a Hugging Face repository.
  • spec.model.name: Specifies the model identifier used by the serving runtime.
  • spec.replicas: Specifies the number of model server replicas.
  • spec.router.scheduler: Specifies the EPP scheduler configuration. The scheduler routes requests to the optimal replica based on queue depth and cache utilization.
  • spec.router.route: Specifies the route configuration. An empty object uses default settings.
  • spec.router.gateway: Specifies the gateway configuration. An empty object uses default settings.
  • spec.template.containers[].resources: Specifies the CPU, memory, and GPU resource requests and limits for the model server container.

When you need to serve multiple models from shared infrastructure, a multi-model deployment places each model behind its own LLMInferenceService resource while sharing the same GPU nodes.

A multi-pool deployment creates multiple independent model-serving pools, each with its own Endpoint Picker (EPP) scheduler, route, and gateway. You can use this pattern to isolate workloads across separate pools or to serve different models on shared cluster infrastructure. This deployment pattern for Distributed Inference with llm-d suits workload isolation and multi-model serving use cases.

The CRs configure the following components:

  • Two separate LLMInferenceService resources: multi-pool-a and multi-pool-b, each operating as an independent serving pool.
  • One replica per pool serving the Qwen/Qwen3-0.6B model from Hugging Face.
  • An EPP scheduler, route, and gateway per pool for independent request routing.
  • Resource requests and limits for each model server container, including one NVIDIA GPU per pool.

Multi-pool LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: multi-pool-a
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
---
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: multi-pool-b
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"

where:

  • metadata.name: Specifies a unique name for each LLMInferenceService resource. Each pool requires a distinct name to create independent serving endpoints.
  • spec.model.uri: Specifies the model location. The hf:// prefix indicates a Hugging Face repository.
  • spec.model.name: Specifies the model identifier used by the serving runtime.
  • spec.replicas: Specifies the number of model server replicas per pool.
  • spec.router.scheduler: Specifies the EPP scheduler configuration for each pool.
  • spec.router.route: Specifies the route configuration.
  • spec.router.gateway: Specifies the gateway configuration. Each pool registers with the Inference Gateway independently.
  • spec.template.containers[].resources: Specifies the CPU, memory, and GPU resource requests and limits for the model server container in each pool.

The following LLMInferenceService custom resource (CR) deploys a model with prefix cache-aware routing enabled. This deployment pattern uses the precise-prefix-cache-producer and prefix-cache-scorer EPP plugins to route requests with similar prompt prefixes to the same replica, increasing KV cache hit rates and reducing time-to-first-token latency.

The CR configures the following components:

  • Two replicas serving the Qwen/Qwen3-0.6B model from Hugging Face.
  • An inline EndpointPickerConfig with four scheduling plugins: precise-prefix-cache-producer, prefix-cache-scorer, queue-scorer, and kv-cache-utilization-scorer.
  • A scheduling profile that weights prefix cache scoring higher than queue and KV cache utilization scoring.
  • vLLM arguments that enable prefix caching with SHA-256 hashing and KV cache event publishing over ZMQ.
Note

Replace the spec.model.uri and spec.model.name values with your model. Update the kv-events-config topic string in VLLM_ADDITIONAL_ARGS to match your model name. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

Prefix cache-aware routing LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: cache-aware
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 2
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
      config:
        inline:
          apiVersion: llm-d.ai/v1alpha1
          kind: EndpointPickerConfig
          plugins:
            - type: precise-prefix-cache-producer
              parameters:
                kvEventsConfig:
                  zmqEndpoint: "tcp://*:5557"
                indexerConfig:
                  kvBlockIndexConfig:
                    enableMetrics: true
                    metricsLoggingInterval: 60000000000
            - type: prefix-cache-scorer
              parameters:
                prefixMatchInfoProducerName: precise-prefix-cache-producer
            - type: queue-scorer
            - type: kv-cache-utilization-scorer
          schedulingProfiles:
            - name: default
              plugins:
                - pluginRef: queue-scorer
                  weight: 2
                - pluginRef: kv-cache-utilization-scorer
                  weight: 2
                - pluginRef: prefix-cache-scorer
                  weight: 3
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS
            value: "--prefix-caching-hash-algo sha256 --block-size 16 --kv-events-config '{\"enable_kv_cache_events\":true,\"publisher\":\"zmq\",\"endpoint\":\"tcp://{{ ChildName .ObjectMeta.Name `-epp-service` }}:5557\",\"topic\":\"kv@${POD_IP}@Qwen/Qwen3-0.6B\"}'"
          - name: POD_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: PYTHONHASHSEED
            value: "42"
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 120
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

  • spec.replicas: Specifies the number of model server replicas. Prefix cache-aware routing requires at least 2 replicas to demonstrate routing benefits.
  • spec.router.scheduler.config.inline: Specifies an inline EndpointPickerConfig resource that configures the EPP scheduling plugins and profiles.
  • precise-prefix-cache-producer: Specifies the plugin that tracks KV cache block state across replicas by consuming cache events published by vLLM over ZMQ.
  • prefix-cache-scorer: Specifies the plugin that scores replicas based on how much of the request’s prompt prefix is already cached in each replica’s KV cache.
  • schedulingProfiles: Specifies the scoring weights for each plugin. In this example, prefix-cache-scorer has a weight of 3, while queue-scorer and kv-cache-utilization-scorer each have a weight of 2.
  • VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --prefix-caching-hash-algo, --block-size, and --kv-events-config flags enable prefix caching and KV cache event publishing.
  • PYTHONHASHSEED: Specifies a fixed Python hash seed to ensure consistent prefix hashing across replicas.

10.5. LLMInferenceService CR for flow control
Copy link

Flow control allows the endpoint picker (EPP) to queue requests when all model server replicas are saturated, rather than dropping or overloading them. When a replica becomes available, the EPP dequeues and forwards the next request. You can enable flow control on any deployment pattern by setting the flowControl feature gate in the EndpointPickerConfig resource.

The following LLMInferenceService custom resource (CR) deploys a model with flow control enabled.

The CR configures the following components:

  • A single replica serving the Qwen/Qwen3-0.6B model from Hugging Face.
  • The flowControl feature gate enabled in the inline EndpointPickerConfig.
  • A default request time-to-live (TTL) of 30 seconds. Requests that remain queued longer than this TTL are rejected.
  • Scheduling plugins for queue scoring, KV cache utilization scoring, and prefix cache scoring.
Note

Replace the spec.model.uri and spec.model.name values with your model. Adjust the defaultRequestTTL value based on your latency requirements. A shorter TTL rejects requests faster under load; a longer TTL allows more time for replicas to become available. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

Flow control LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: flow-control
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
      config:
        inline:
          apiVersion: llm-d.ai/v1alpha1
          kind: EndpointPickerConfig
          featureGates:
          - flowControl
          plugins:
            - type: queue-scorer
            - type: kv-cache-utilization-scorer
            - type: prefix-cache-scorer
          flowControl:
            defaultRequestTTL: 30s
          schedulingProfiles:
            - name: default
              plugins:
                - pluginRef: queue-scorer
                  weight: 2
                - pluginRef: kv-cache-utilization-scorer
                  weight: 2
                - pluginRef: prefix-cache-scorer
                  weight: 3
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS
            value: "--max-num-seqs 2"
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

  • spec.router.scheduler.config.inline.featureGates: Specifies the feature gates to enable. The flowControl gate activates request queuing in the EPP.
  • spec.router.scheduler.config.inline.flowControl.defaultRequestTTL: Specifies the maximum time a request can remain in the queue before the EPP rejects it.
  • VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --max-num-seqs 2 flag limits the maximum number of concurrent sequences per replica, which makes it easier to trigger flow control queuing for testing purposes.

Token-based flow control extends the default flow control behavior by detecting replica saturation based on the number of in-flight tokens rather than the number of concurrent connections. This approach prevents overloading replicas that are processing long sequences, because a single request with many tokens consumes more capacity than a short request.

The following LLMInferenceService custom resource (CR) deploys a model with token-based flow control enabled.

Token-based flow control LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: flow-control-tokens
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
      config:
        inline:
          apiVersion: llm-d.ai/v1alpha1
          kind: EndpointPickerConfig
          featureGates:
          - flowControl
          plugins:
            - type: queue-scorer
            - type: kv-cache-utilization-scorer
            - type: prefix-cache-scorer
            - name: concurrency-detector
              type: concurrency-detector
              parameters:
                concurrencyMode: "tokens"
                maxTokenConcurrency: 200
            - name: inflight-load-producer
              type: inflight-load-producer
              parameters:
                addEstimatedOutputTokens: true
          flowControl:
            defaultRequestTTL: 30s
            saturationDetector:
              pluginRef: concurrency-detector
          schedulingProfiles:
            - name: default
              plugins:
                - pluginRef: queue-scorer
                  weight: 2
                - pluginRef: kv-cache-utilization-scorer
                  weight: 2
                - pluginRef: prefix-cache-scorer
                  weight: 3
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS
            value: "--max-num-seqs 2"
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

  • spec.router.scheduler.config.inline.featureGates: Specifies the feature gates to enable. The flowControl gate activates request queuing in the EPP.
  • concurrency-detector.parameters.concurrencyMode: Specifies the concurrency detection mode.
  • concurrency-detector.parameters.maxTokenConcurrency: Specifies the maximum number of concurrent tokens per replica before the EPP considers the replica saturated and begins queuing requests.
  • inflight-load-producer.parameters.addEstimatedOutputTokens: Specifies whether to include estimated output tokens when calculating in-flight load.
  • flowControl.saturationDetector.pluginRef: Specifies the plugin that determines when a replica is saturated. The concurrency-detector reference connects token-based detection to the flow control system.
  • flowControl.defaultRequestTTL: Specifies the maximum time a request can remain in the queue before the EPP rejects it.
  • VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --max-num-seqs 2 flag limits the maximum number of concurrent sequences per replica, which makes it easier to trigger flow control queuing for testing purposes.

Low-Rank Adaptation (LoRA) adapter serving lets you load lightweight fine-tuned model layers on top of a base model without duplicating the full model weights. You can serve multiple specialized model variants from a single base model deployment, reducing GPU memory consumption compared to deploying each variant as a separate model.

Important

LoRA adapter serving is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

The following LLMInferenceService custom resource (CR) examples deploy a base model with one or more Low-Rank Adaptation (LoRA) adapters.

Single LoRA adapter

The following CR deploys the facebook/opt-125m base model with a single LoRA adapter for SQL generation.

Note

Replace the spec.model.uri and spec.model.name values with your base model. Replace the adapter name and uri values with your LoRA adapter. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

LoRA adapter serving LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: lora-single
spec:
  replicas: 1
  model:
    uri: hf://facebook/opt-125m
    name: facebook/opt-125m
    lora:
      adapters:
        - name: sql-adapter
          uri: hf://edbeeching/opt-125m-lora
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

  • spec.model.uri: Specifies the base model location. The hf:// prefix indicates a Hugging Face repository.
  • spec.model.lora.adapters: Specifies the list of LoRA adapters to load on top of the base model.
  • spec.model.lora.adapters[].name: Specifies the name of the adapter. Clients use this name to select the adapter in inference requests.
  • spec.model.lora.adapters[].uri: Specifies the adapter location in Hugging Face.
Multiple LoRA adapters

The following CR deploys the same base model with two LoRA adapters and sets limits on the maximum number of adapters and the maximum LoRA rank. 

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: lora-multi
spec:
  replicas: 1
  model:
    uri: hf://facebook/opt-125m
    name: facebook/opt-125m
    lora:
      adapters:
        - name: sql-adapter
          uri: hf://edbeeching/opt-125m-lora
        - name: code-adapter
          uri: hf://velaa/opt-125m-lora-finetuned-mnli
      maxAdapters: 2
      maxRank: 64
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

  • spec.model.lora.adapters: Specifies the list of LoRA adapters. In this example, two adapters are configured: sql-adapter and code-adapter.
  • spec.model.lora.maxAdapters: Specifies the maximum number of LoRA adapters that can be loaded simultaneously. Set this value based on your available GPU memory.
  • spec.model.lora.maxRank: Specifies the maximum LoRA rank allowed for adapters. Higher ranks use more GPU memory but can capture more complex fine-tuning patterns.

The following LLMInferenceService custom resource (CR) deploys a model with prefill/decode disaggregation on GPU nodes with network connectivity between prefill and decode pods. Single-node and multi-node deployments require RDMA-capable network interfaces such as RoCE v2 or InfiniBand for high-throughput, low-latency KV cache transfer between prefill and decode pods.

Important

Prefill/decode disaggregation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The CR configures the following components:

  • One decode replica and two prefill replicas serving the Qwen/Qwen3-0.6B model from Hugging Face.
  • NIXL for KV cache transfer between prefill and decode workers over TCP.
  • UCX transport configuration for inter-worker communication.
  • The EPP scheduler for routing requests to decode replicas.

Prefill/decode disaggregation LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: pd
spec:
  replicas: 1
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  router:
    route: {}
    gateway: {}
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_NIXL_SIDE_CHANNEL_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: VLLM_ADDITIONAL_ARGS
            value: "--kv_transfer_config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\"}'"
          - name: UCX_TLS
            value: "tcp,sm,self,cuda_copy,cuda_ipc"
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 120
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5
  prefill:
    replicas: 2
    template:
      imagePullSecrets:
      - name: rhai-pull-secret
      containers:
        - name: main
          env:
            - name: VLLM_NIXL_SIDE_CHANNEL_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: VLLM_ADDITIONAL_ARGS
              value: "--kv_transfer_config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\"}'"
            - name: UCX_TLS
              value: "tcp,sm,self,cuda_copy,cuda_ipc"
          resources:
            limits:
              cpu: '4'
              memory: 32Gi
              nvidia.com/gpu: "1"
            requests:
              cpu: '2'
              memory: 16Gi
              nvidia.com/gpu: "1"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
              scheme: HTTPS
            initialDelaySeconds: 120
            periodSeconds: 30
            timeoutSeconds: 30
            failureThreshold: 5

where:

  • spec.replicas: Specifies the number of decode replicas. Decode replicas generate output tokens one at a time while reading from the KV cache.
  • spec.prefill: Specifies the prefill pool configuration. The prefill pool is separate from the main decode pool.
  • spec.prefill.replicas: Specifies the number of prefill replicas. Prefill replicas process the full input context in parallel.
  • VLLM_NIXL_SIDE_CHANNEL_HOST: Specifies the pod IP address for NIXL side-channel communication. This value is populated automatically from the pod’s status.podIP field.
  • VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --kv_transfer_config flag configures the NIXL connector for KV cache transfer between prefill and decode workers.
  • UCX_TLS: Specifies the UCX transport layers for inter-worker communication. The value tcp,sm,self,cuda_copy,cuda_ipc enables TCP transport alongside shared memory and CUDA transports.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top