SupportConsoleDevelopersStart a trialContact

Chapter 3. Deploying DiffusionGemma models

You can deploy DiffusionGemma, a discrete diffusion language model, on the model serving platform by using the vLLM ServingRuntime for KServe. DiffusionGemma uses block diffusion to generate text in parallel, which provides higher per-request generation throughput compared to autoregressive models.

DiffusionGemma is a discrete diffusion language model that uses block diffusion to generate text in parallel instead of sequentially. You can deploy DiffusionGemma on OpenShift AI to achieve higher per-request generation throughput compared to autoregressive models.

3.1.1. How block diffusion works
Copy link

Traditional autoregressive language models generate tokens one at a time in a sequential process where each new token depends on all previously generated tokens. This sequential approach limits generation speed because the model must wait for each token before producing the next one.

DiffusionGemma uses a different approach called block diffusion. Instead of generating tokens sequentially, the model initializes a fixed 256-token canvas with masked tokens, then iteratively denoises the entire canvas in parallel. Over multiple refinement steps, the model replaces the masked tokens with coherent text. Because the model processes all 256 tokens simultaneously, block diffusion achieves approximately 6x higher per-request generation throughput compared to autoregressive models of similar size.

This throughput improvement requires a tradeoff in time-to-first-token, or TTFT. DiffusionGemma requires approximately 10x higher TTFT than an equivalent autoregressive model because the initial canvas must be fully denoised before any tokens are available. For example, on an H100 GPU, DiffusionGemma achieves a TTFT of approximately 489 ms compared to approximately 53 ms for autoregressive Gemma 4.

DiffusionGemma is a 26 billion parameter model built on the Gemma 4 Mixture-of-Experts architecture, with 3.8 billion active parameters per token. The model uses the standard OpenAI-compatible /v1/chat/completions API endpoint. The diffusion decoding process is transparent to API consumers and does not require changes to client applications.

3.1.2. Available model variants
Copy link

The following table lists the DiffusionGemma model variants that you can deploy on OpenShift AI.

Table 3.1. DiffusionGemma model variants
ModelQuantizationMinimum vRAMValidated GPU

RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic

FP8

31.3 GB

1x A100-80 or H100

3.1.3. Known limitations
Copy link

DiffusionGemma has the following limitations:

  • Single-GPU only: Tensor parallelism and pipeline parallelism with more than one GPU are not supported. You must deploy the model on a single GPU with sufficient vRAM.
  • Maximum 4 concurrent sequences: You must set the --max-num-seqs argument to 4 or fewer. Values greater than 4 cause out-of-memory errors due to diffusion state buffer memory requirements.
  • Higher time-to-first-token: DiffusionGemma requires approximately 10x higher TTFT than an equivalent autoregressive model because the initial denoising process must complete before tokens are available.

3.2. Deploy a DiffusionGemma model
Copy link

You can deploy a DiffusionGemma discrete diffusion language model on OpenShift AI by using the vLLM ServingRuntime for KServe. DiffusionGemma requires specific serving runtime arguments to enable diffusion decoding and manage memory constraints.

Prerequisites

  • You have logged in to OpenShift AI as a user with OpenShift AI administrator privileges.
  • You have downloaded the DiffusionGemma FP8 model to your object storage.
  • You have enabled the vLLM ServingRuntime for KServe runtime.
  • You have enabled GPU support in OpenShift AI and have installed and configured the Node Feature Discovery Operator on your cluster. For more information, see Installing the Node Feature Discovery Operator and Enabling NVIDIA GPUs.
  • You have an NVIDIA GPU with at least 32 GB of vRAM. DiffusionGemma is validated on A100-80 and H100 GPUs.

Procedure

  1. Follow the steps to deploy a model as described in Deploying models on the model serving platform.
  2. In the Serving runtime field, select vLLM ServingRuntime for KServe.

  3. Add the following arguments under Additional serving runtime arguments in the Configuration parameters section: 

    --diffusion-config={"canvas_length": 256}
--hf-overrides={"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}
--max-num-seqs=4
--generation-config=vllm

    where:

    • --diffusion-config={"canvas_length": 256}: Enables diffusion decoding with 256-token canvas blocks.
    • --hf-overrides={"diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1}: Configures the entropy-bound denoising sampler with a threshold of 0.1.
    • --max-num-seqs=4: Limits concurrent sequences to 4. Values above 4 cause out-of-memory errors due to diffusion state buffer memory requirements.
    • --generation-config=vllm: Overrides the checkpoint default 256-token max_tokens cap. Without this argument, model responses are silently truncated to 256 tokens.

  4. Add the following environment variable under Additional serving runtime environment variables in the Configuration parameters section: 

    VLLM_USE_V2_MODEL_RUNNER=1
  5. Click Deploy.

Verification

  • Confirm that the deployed model is shown on the Deployments tab for the project, and on the Deployments page of the dashboard with a checkmark in the Status column.

  • Run the following command to verify API requests to your deployed model. In the following command, replace <inference_endpoint_url> with the URL of your inference endpoint. Replace <model_name> with the name of your deployed DiffusionGemma model. 

    $ curl -X POST \
    "https://<inference_endpoint_url>:443/v1/chat/completions" \
    -H "accept: application/json" \
    -H "Content-Type: application/json" \
    -d '{
    "model": "<model_name>",
    "messages": [{"role": "user", "content": "What is discrete diffusion?"}],
    "max_tokens": 512
    }'

Troubleshooting

  • If responses are truncated to approximately 256 tokens, verify that --generation-config=vllm is included in the serving runtime arguments.
  • If the model server crashes with an out-of-memory error, verify that --max-num-seqs is set to 4 or fewer.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top