Chapter 3. Version 3.5.0-ea.2 release notes

Structured output generation on IBM Z with llguidance backend now works correctly.

Previously, when using the llguidance structured decoding backend on IBM Z (s390x), JSON Schema-constrained generation produced invalid output or became stuck generating whitespace indefinitely. This issue is now resolved.

Sarvam-105B model now loads successfully.

Previously, the RedHatAI/sarvam-105b-FP8-Dynamic model failed to load due to a RoPE validation error caused by a Transformers v5 incompatibility. This issue is now resolved.

Prithvi TerraTorch model warmup fails on arm64 GH200 due to tensor size limit.

When deploying the ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11 model on arm64 NVIDIA GH200 accelerators, the vLLM engine core fails during warmup with the error "upsample_bilinear2d_nhwc only supports output tensors with less than INT_MAX elements". A PyTorch CUDA kernel limitation on arm64 causes this issue when the TerraTorch UPerNet decoder warmup tensor exceeds the signed 32-bit integer maximum.

To work around this issue, deploy the model with limited concurrency:

--max-num-seqs=8 --max-num-batched-tokens=8

This configuration does not affect performance for geospatial models because each request uses only one token.

DeepSeek-V4-Flash-NVFP4-FP8 model crashes during initialization.

The RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8 quantized model fails to load with a KeyError for scale_fmt in the quantization configuration. The DeepseekV4Attention initialization expects a scale_fmt key that is not present in the model configuration.

Mistral Small 3.1 tool calling returns no tool calls with tool_choice=auto.

When using the Mistral Small 3.1 model with tool calling and tool_choice=auto, the model does not return tool calls in the response.

Qwen2.5-VL FP8 tool calling fails because the model does not support tools.

The Qwen2.5-VL FP8 model fails tool calling smoke tests because the model architecture does not support tool calling functionality.

Mistral Small FP8-dynamic fails to start due to missing backend attribute.

The Mistral Small FP8-dynamic model fails to start with an error indicating that CachedMistralCommonBackend is missing the is_fast attribute.

Gemma 4 12B model evaluation crashes due to batch token limit.

When running LM Evaluation Harness on the Gemma 4 12B model, the evaluation crashes with an error indicating that max_num_batched_tokens is too small for the prefix-LM multimodal model.

To work around this issue, explicitly set --max-num-batched-tokens to any value greater than 2496:

--max-num-batched-tokens 4096

Granite 4.0-h-small model crashes on first inference with ROCm.

When deploying the ibm-granite/granite-4.0-h-small model on AMD ROCm accelerators, the vLLM server crashes during the first inference request with a RuntimeError: PassManager::run failed error. This occurs because the Triton JIT compiler fails to compile Mamba chunk_scan kernels on the first run. After the pod restarts, the model works correctly due to the warm compilation cache.

Qwen 3.5 and 3.6 models fail on CPU deployments.

Qwen 3.5 and 3.6 models fail to run correctly on CPU-only deployments due to an upstream vLLM issue with GDN attention operations.

Intel Gaudi deployments fail due to habana-torch-plugin version rejection.

When deploying models on Intel Gaudi accelerators, the vllm-gaudi configuration validation rejects the habana-torch-plugin version 1.23.0.post1 because it does not conform to PEP 440 versioning requirements.

Prefix caching with models requiring transformers 5.x crashes the inference scheduler.

Deploying models whose architecture was introduced in transformers 5.0 or later with prefix caching enabled (precise-prefix-cache-scorer) causes the inference scheduler to crash. The tokenizer sidecar (odh-llm-d-kv-cache-rhel9) ships transformers 4.57.6, which does not recognize architecture types added in transformers 5.x. Affected models include Gemma 4, GLM-5, Qwen3.5 MoE, and others added after the 4.x series.

To work around this issue, disable precise-prefix-cache-scorer in the EndpointPickerConfig when using these models.

This issue is fixed in 3.5 stable. The UDS tokenizer sidecar is replaced by a vLLM render sidecar that stays in sync with the serving image’s transformers version.