Chapter 4. Intelligent inference scheduling architecture

You can use intelligent inference scheduling in Distributed Inference with llm-d to route inference requests to the optimal model server replica based on real-time metrics such as queue depth, KV cache utilization, and prefix cache state. Intelligent scheduling reduces inference latency by up to 2x and increases throughput by up to 3x compared to round-robin load balancing by making cache-aware, load-aware routing decisions.

The Distributed Inference with llm-d inference stack uses a four-layer architecture that separates concerns between networking, scheduling intelligence, model serving, and observability.

Gateway layer

The Gateway layer handles ingress traffic using the Kubernetes Gateway API with a supported data plane such as Istio. The Gateway API Inference Extension integrates the Endpoint Picker (EPP) into the request path by using the Envoy external processing (ext-proc) protocol. When a request arrives at the inference gateway, Envoy forwards it to the Endpoint Picker for intelligent routing before delivering it to the selected model server.

The Gateway layer provides:

TLS termination and certificate management, optionally through cert-manager
HTTP and gRPC protocol support for inference requests, where gRPC is used for the ext-proc communication between the gateway and the EPP
Integration with the Kubernetes Gateway API Inference Extension specification

Scheduling layer

The Endpoint Picker (EPP) is a request-path component that makes workload placement decisions for every incoming inference request. For each incoming inference request, the EPP evaluates all available model server replicas and selects the optimal target by using a configurable set of scheduling plugins.

The EPP uses a scheduling lifecycle that includes filter, score, and pick plugins:

Filter plugins eliminate unsuitable replicas.
Score plugins assign weighted scores to remaining replicas based on metrics such as queue depth, prefix cache affinity, and predicted latency.
Picker plugins select the final target from scored replicas.

You can customize the scheduling behavior by configuring scheduling profiles that define which plugins are active and how their scores are weighted. The EPP supports automatic plugin activation, which activates a default set of plugins when no explicit configuration is provided.

Data plane layer

The Data Plane layer consists of vLLM model server pods that run inference requests. Each vLLM instance exposes Prometheus metrics on port 8000 at the /metrics endpoint, providing real-time data about queue depth, KV cache utilization, prefix cache state, and token throughput.

The EPP continuously monitors these metrics to maintain an up-to-date view of each replica’s state, enabling informed scheduling decisions.

In multi-node deployments, LeaderWorkerSet (LWS) manages the lifecycle of distributed inference workloads that span multiple GPU nodes, such as Mixture of Experts (MoE) models with expert parallelism.

Observability layer

The Observability layer collects and exposes Prometheus-compatible metrics from all inference components. On OpenShift, the controller automatically creates PodMonitor and ServiceMonitor resources to scrape metrics from vLLM engines and the EPP. On other Kubernetes platforms, you must configure Prometheus scrape targets manually.

You can use these metrics to build dashboards, configure alerts, verify scheduling effectiveness, and plan fleet capacity.

Chapter 4. Intelligent inference scheduling architecture

Learn

Try, buy, & sell

Communities

About Red Hat

Making open source more inclusive

About Red Hat Documentation

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links