Chapter 7. Disaggregated and aggregated inference serving modes

In aggregated mode, each model-serving pod handles both inference phases: prefill and decode. The prefill phase processes the full input prompt in parallel to populate the key-value (KV) cache. The decode phase generates output tokens one at a time by using the cached data.

All pods in the deployment share the same resource profile, and the inference scheduler routes each request to a single pod that performs both phases sequentially.

Aggregated mode is simpler to operate and works well for workloads with uniform resource demands, small models, or short input sequences where the prefill phase does not dominate GPU utilization.

In disaggregated mode, the prefill and decode phases run on independent pod pools with separate replica counts and resource allocations. Prefill pods process incoming prompts and transfer the resulting KV cache data to decode pods over RDMA-capable networking by using the NIXL connector. Decode pods then generate output tokens from the transferred KV cache without repeating the prefill computation.

Disaggregated mode creates two separate Kubernetes Deployment resources: one for prefill pods and one for decode pods.

Disaggregated mode enables independent scaling of each phase. You can configure different replica counts and tensor parallelism degrees for prefill and decode pods to match their resource profiles. For example, you can run more prefill replicas at lower tensor parallelism for compute throughput, and fewer decode replicas at higher tensor parallelism for memory bandwidth. Both pod pools must use the same GPU type because KV cache layout must be compatible between prefill and decode pods.

By eliminating interference between the compute-intensive prefill phase and the memory-bandwidth-intensive decode phase, disaggregated mode improves GPU utilization and reduces inter-token latency (ITL). For workloads with latency-sensitive service level objectives, disaggregated mode can significantly reduce tail latency. However, because fewer total resources are dedicated to prefill processing, time-to-first-token (TTFT) typically increases compared to aggregated mode.