SupportConsoleDevelopersStart a trialContact

Chapter 7. Known issues

Understand how newly identified and previously known issues might affect your use of Red Hat OpenShift AI, and how to work around them.

7.1. Issues discovered at version 3.5 GA
Copy link

RHOAIENG-87625 - Kubeflow Trainer v2 is not compatible with Red Hat Build of Kueue 1.4 or later

The Kubeflow Trainer v2 component in OpenShift AI 3.5 is not compatible with Red Hat Build of Kueue (RHBoK) 1.4 or later. Kubeflow Trainer v2 continues to function, but the integration between Kubeflow Trainer v2 and Kueue is broken if RHBoK is upgraded to version 1.4 or later.

Workaround

If you use Kubeflow Trainer v2 with Kueue, keep RHBoK at version 1.3 or earlier and do not upgrade to RHBoK 1.4 or later.

Note

OpenShift AI 3.5 currently requires RHBoK 1.3 for Kubeflow Trainer v2 integration with Kueue, even though RHBoK 1.3 is no longer supported. In addition, RHBoK 1.3 is not available with OpenShift Container Platform 4.22.

RHOAIENG-82694 - MCP Lifecycle Operator memory consumption might cause the operator to fail

The MCP Lifecycle Operator watches all ConfigMaps and Secrets cluster-wide by using full structured informers, loading complete object data into the operator’s in-memory cache. As a consequence, memory consumption grows proportionally to the number of cluster-wide Secrets and ConfigMaps, causing the operator to fail with an out-of-memory error on large clusters even when no MCP servers are deployed.

Workaround
No known workaround exists.

RHOAIENG-85220 - “Legacy deployment” wording is misleading for generative InferenceService deployments

In the OpenShift AI model deployment UI, generative model deployments that use an InferenceService with a ServingRuntime display a Legacy deployment label. This wording might incorrectly suggest that this deployment method is being removed or is no longer supported.

Support for generative deployments that use an InferenceService with a ServingRuntime has not changed. However, for generative workloads, the recommended approach is to use LLMInferenceService, which provides additional capabilities and enables models to transition to the Models-as-a-Service (MaaS) infrastructure.

Predictive workloads are not affected by this wording and continue to use InferenceService with ServingRuntime deployments.

Workaround
No workaround is required. You can continue to use InferenceService with ServingRuntime deployments. Improved wording in the model deployment UI is planned for a future release.

RHOAIENG-66855 - Distributed Inference with llm-d deployment through the dashboard selects CUDA image instead of ROCm for AMD hardware profiles

When you deploy a model through the OpenShift AI dashboard by using distributed inference with Distributed Inference with llm-d with AMD hardware profiles, such as mi300x, the deployment incorrectly uses the CUDA runtime image instead of the ROCm image. The LLMInferenceService controller does not detect AMD GPU resources and instead assigns the default template, which results in the CUDA image being used instead of the ROCm-specific image.

As a consequence, the pod fails to start on AMD hardware.

Workaround
Deploy the LLMInferenceService by using a YAML manifest with the ROCm image explicitly specified in the container spec. Alternatively, for non-Distributed Inference with llm-d vLLM deployments, manually select the "vLLM AMD GPU ServingRuntime for KServe" option in the dashboard, which correctly uses the AMD-specific image.

7.2. Issues discovered at version 3.5 EA2
Copy link

RHOAIENG-76586 - Rate limiting stops working with Red Hat Connectivity Link 1.4.x

When you use Red Hat Connectivity Link (RHCL) 1.4.0 or 1.4.1 with batch gateway deployments, rate limiting silently stops functioning. RHCL 1.4 changed how the wasm plugin is injected, which breaks the automatic passing of authentication identity data into the wasm plugin context. As a result, rate limit counters that use auth.identity.user.username fail silently, and requests are never rate limited. Additionally, gateway pods can crash with OOMKilled errors because the RHCL 1.4 wasm plugin requires more memory to compile than the default 1Gi limit.

Workaround

Use RHCL 1.3.5 for batch gateway deployments. RHCL 1.3.5 works without additional configuration.

If your cluster already has RHCL 1.4.x installed, apply both of the following workarounds:

  1. Increase gateway pod memory to 2Gi. Create a ConfigMap with a memory override and link it to the gateway by using infrastructure.parametersRef: 

    $ oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: <gateway-name>-proxy-config
  namespace: openshift-ingress
data:
  deployment: |
    spec:
      template:
        spec:
          containers:
          - name: istio-proxy
            resources:
              limits:
                memory: 2Gi
EOF
     
    $ oc patch gateway <gateway-name> -n openshift-ingress --type=merge \
  -p '{"spec":{"infrastructure":{"parametersRef":{"group":"","kind":"ConfigMap","name":"<gateway-name>-proxy-config"}}}}'

    where:

<gateway-name>

Specifies the name of the gateway, for example openshift-ai-inference or batch-internal-gateway.

  1. Add an identity filter to your AuthPolicy to populate the wasm plugin identity context. Add a response.success.filters.identity section to your AuthPolicy: 

    spec:
  rules:
    response:
      success:
        filters:
          identity:
            json:
              properties:
                userid:
                  expression: auth.identity.user.username
                user:
                  expression: auth.identity.user
            metrics: false
            priority: 0

    The filters.identity section tells Authorino to write the authenticated user identity into the wasm plugin context. Without it, rate limit counters that use auth.identity.user.username fail and rate limiting is silently skipped.

  2. If you have a batch-route AuthPolicy with a RateLimitPolicy counter that uses auth.identity.user.username, add the same identity filter to the batch-route AuthPolicy: 

    $ BATCH_NS=batch-api
$ oc apply -f - <<EOF
apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  name: batch-route-auth
  namespace: ${BATCH_NS}
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: batch-route
  rules:
    authentication:
      kubernetes-user:
        kubernetesTokenReview:
          audiences:
          - https://kubernetes.default.svc
        metrics: false
        priority: 0
    response:
      success:
        filters:
          identity:
            json:
              properties:
                userid:
                  expression: auth.identity.user.username
                user:
                  expression: auth.identity.user
            metrics: false
            priority: 0
EOF

    After applying all steps, verify that rate limiting works: 

    $ GW_HOSTNAME=$(oc get gateway openshift-ai-inference -n openshift-ingress \
    -o jsonpath='{.spec.listeners[0].hostname}')
$ TOKEN=$(oc create token test-authorized-sa -n llm \
    --audience=https://kubernetes.default.svc --duration=10m)
$ for i in $(seq 1 20); do
    code=$(curl -sk -o /dev/null -w '%{http_code}' \
        "https://${GW_HOSTNAME}/llm/facebook-opt-125m/v1/chat/completions" \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer ${TOKEN}" \
        -d '{"model":"facebook/opt-125m","messages":[{"role":"user","content":"Hi"}],"max_tokens":100}')
    echo "Request $i: $code"
    [ "$code" = "429" ] && echo "Rate limiting is working!" && break
done

    If rate limiting is working, one of the requests returns a 429 status code.

RHOAIENG-73389 - Prefix caching causes inference scheduler crash for models requiring transformers 5.x

The tokenizer sidecar odh-llm-d-kv-cache-rhel9 ships transformers 4.57.6, which does not recognize model architecture types added in transformers 5.x. As a consequence, deploying these models with precise-prefix-cache-scorer enabled causes the inference scheduler to crash. Affected models include Gemma 4, GLM-5, Qwen3.5 MoE, and other models with architecture types added after the transformers 4.x series.

Workaround
Disable precise-prefix-cache-scorer in the EndpointPickerConfig when using these models.

7.3. Issues discovered at version 3.5 EA1
Copy link

RHOAIENG-66068 - The OpenShift AI dashboard only supports an EvalHub instance in the redhat-ods-applications namespace

The Backend-for-Frontend (BFF) service always looks for the MLflow multi-tenant instance in its own redhat-ods-applications, regardless of where the cluster administrator has deployed it. As a consequence, the OpenShift AI dashboard reports that evaluations are not enabled when the multi-tenant instance is hosted in a different namespace.

Workaround
Deploy the MLflow multi-tenant instance in the redhat-ods-applications. As a result, the BFF service correctly detects the instance and the evaluations feature is available in the dashboard. Note that others instances will work, but will not be discoverable from the OpenShift AI dashboard.

RHOAIENG-67534 - A new evaluation run fails in the OpenShift AI dashboard

If the MLflow custom resource (CR) is created after the Evaluations CR, the workspaces_enabled setting is set to false. As a result, creating a new evaluation run in the OpenShift AI dashboard fails with an INVALID_PARAMETER_VALUE error: "Workspace context is required for this request."

Workaround
Create the MLflow CR before the Evaluations CR is created. This ensures the workspaces_enabled setting is correctly set to true, and evaluation runs can be created successfully.

AIPCC-18235 - Structured output (JSON Schema) generation fails on IBM Z (s390x) with llguidance backend

When you use the llguidance structured decoding backend on IBM Z (s390x), JSON schema-constrained generation may produce invalid output or become stuck generating whitespace indefinitely.

Workaround
A fix is available in llguidance version 1.7.0 and later. Update your wheel from version 1.3.0 to at least version 1.7.0 for this fix.

AIPCC-17927 - vLLM crashes when multiple requests are inflight with structured outputs

When you send multiple inference requests in parallel to a vLLM-based inference server and at least one request includes structured output, the service stops responding, causing the pod to fail. As a result, concurrent workloads that use structured outputs do not function as expected.

Workaround

To prevent the service from failing, apply one of the following workarounds:

  • Process requests sequentially instead of sending multiple parallel requests that include structured output in the same batch.
  • Exclude structured output requests when you run concurrent workloads.

7.4. Issues discovered at version 3.4 GA
Copy link

RHOAIENG-83207 - Secure Models-as-a-Service gateway requires namespace labeling for redhat-ods-applications

When you deploy a secure Models-as-a-Service (MaaS) gateway that restricts route attachment by namespace selector, you must also label the redhat-ods-applications namespace. OpenShift AI creates an internal HTTPRoute in that namespace to serve the MaaS API. Without the label, the gateway rejects the route and the API Keys page in the dashboard displays "Error loading components."

Workaround

Label the namespace so that the gateway accepts the route: 

$ oc label namespace redhat-ods-applications maas-gateway-access=true

RHOAIENG-80360 - Insecure Models-as-a-Service gateway configuration can allow unauthorized route hijacking

When creating the Gateway for Models-as-a-Service (MaaS) or Distributed Inference with llm-d, do not use default settings such as allowedRoutes.namespaces.from: All. Such a configuration permits any namespace on the cluster to attach HTTPRoutes to the shared model-serving Gateway. Because the Kubernetes Gateway API prioritizes exact path matches over the prefix matches used by MaaS, a user with standard namespace-level permissions can create an HTTPRoute that silently intercepts and hijacks traffic intended for a legitimate model endpoint. This allows unauthorized interception of sensitive data, such as API keys, user prompts, and model responses, without requiring elevated privileges.

Workaround
For details on how to detect unauthorized routes and apply a secure, label-based namespace configuration, see Insecure Models-as-a-Service gateway configuration can allow unauthorized route hijacking in Red Hat OpenShift AI.

RHOAIENG-67403: Hardware profiles with Node selectors and tolerations are unavailable for workbenches after enabling Kueue

When the Kueue component is set to unmanaged in the DataScienceCluster custom resource, hardware profiles configured with "Node selectors and tolerations" as their workload allocation strategy are not available for selection when you create a workbench. Currently, only hardware profiles configured to use a local queue are available.

INFERENG-6962 - Distributed Inference with llm-d EndpointPicker is bypassed when multiple HTTPRoutes share the same gateway listener

When multiple HTTPRoutes are attached to the same wildcard Gateway listener, Istio aggregates them into a single autogenerated Gateway VirtualService and does not create the per-route ExtProcPerRoute override for the LLMInferenceService. This causes the EndpointPicker to be bypassed entirely. Requests fall back to round-robin routing; prefix cache scoring, load-aware scoring, and all intelligent scheduling are silently disabled.

This behavior is not specific to multiple LLMInferenceServices and is triggered by any HTTPRoute on the same wildcard Gateway listener, such as a token endpoint, echo service, or test route.

You can identify this issue by checking the EndpointPicker logs, which might show no per-request activity, even at verbosity level 6 or 7. Additionally, the gateway ext_proc filter shows cluster_name: "dummy" and request_header_mode: SKIP with no per-route override applied.

This affects Istio 1.26, deployed by openshift-ingress in OSSM 3.3.x and 3.4. The upstream fix is in Istio 1.29. The following issue is related: OSSM-12585.

Workaround
Remove or reassign any non-LLMInferenceService HTTPRoutes from the inference Gateway. Move them to a separate Gateway so the LLMInferenceService HTTPRoute is the only consumer of the wildcard listener.

RHOAIENG-71638 - Models-as-a-Service returns HTTP 500 errors under high concurrent request load

Red Hat Connectivity Link (RHCL) configures the Kuadrant WASM plugin with a default authentication service timeout of 200 ms. Under high concurrent request load, Authorino authentication latency can exceed this threshold when cache misses occur. Because the WASM plugin uses failureMode: deny, timed-out authentication requests result in HTTP 500 or 503 responses to clients, even when model backends are healthy.

This issue affects Red Hat OpenShift AI 3.4 and 3.5.

Workaround

Set AUTH_SERVICE_TIMEOUT to 2s on the RHCL operator Subscription:

  1. Set the Subscription name and namespace for your installation: 

    $ RHCL_NAMESPACE=rh-connectivity-link
$ RHCL_SUBSCRIPTION=rhcl-operator

  2. Patch the Subscription to set AUTH_SERVICE_TIMEOUT to 2s. Use the command that matches your Subscription configuration:

    • If spec.config.env already exists on the Subscription, append the timeout value: 

      $ oc patch subscription "${RHCL_SUBSCRIPTION}" -n "${RHCL_NAMESPACE}" \
  --type='json' \
  -p='[{"op":"add","path":"/spec/config/env/-","value":{"name":"AUTH_SERVICE_TIMEOUT","value":"2s"}}]'

    • If spec.config exists but spec.config.env does not, add the env array: 

      $ oc patch subscription "${RHCL_SUBSCRIPTION}" -n "${RHCL_NAMESPACE}" \
  --type='json' \
  -p='[{"op":"add","path":"/spec/config/env","value":[{"name":"AUTH_SERVICE_TIMEOUT","value":"2s"}]}]'

    • If the Subscription does not have a spec.config section, create it: 

      $ oc patch subscription "${RHCL_SUBSCRIPTION}" -n "${RHCL_NAMESPACE}" \
  --type='json' \
  -p='[{"op":"add","path":"/spec/config","value":{"env":[{"name":"AUTH_SERVICE_TIMEOUT","value":"2s"}]}}]'

  3. After patching the Subscription, wait for the operator to roll out: 

    $ oc wait --for=jsonpath='{.status.state}'=AtLatestKnown \
  subscription/${RHCL_SUBSCRIPTION} -n ${RHCL_NAMESPACE} --timeout=300s
$ oc wait --for=condition=Available --timeout=120s \
  deployment/kuadrant-operator-controller-manager -n ${RHCL_NAMESPACE}
  4. Retest the concurrent workload to verify that HTTP 500 or 503 errors no longer occur.

7.5. Issues discovered at version 3.4 EA2
Copy link

RHOAIENG-58765 - Distributed Inference with llm-d prefill and decode disaggregation fails on FIPS-enabled clusters

Using Distributed Inference with llm-d prefill and decode disaggregation for LLM deployments on FIPS-enabled clusters causes the routing sidecar pod to enter a crash loop, preventing the LLM deployment from functioning. This issue is caused by a runtime image introduced in the 3.4 EA2 release that is not FIPS-compatible.

Workaround
Do not use prefill and decode disaggregation with Distributed Inference with llm-d in Red Hat OpenShift AI 3.4 EA2 on FIPS-enabled clusters. Other features continue to work correctly on FIPS-enabled clusters.

RHOAIENG-57224 - ROCm universal image training produces NaN on MI300X due to torch aotriton 0.11.1 regression

ROCm universal training image (th06) produces NaN values on MI300X due to aotriton 0.11.1 regression in AIPCC-built PyTorch wheel.

Workaround
Use th05 image or set attn_implementation="flash_attention_2".

RHOAIENG-57427 - RAG in Gen AI Playground doesn’t work with default system prompt and model Qwen/Qwen3-14B-AWQ

In Gen AI Playground RAG, the default system prompt might not reliably trigger the knowledge search/tool-calling behavior for some models, so document retrieval is not performed. Due to this, questions about uploaded documents can return answers without using the vector store, resulting in incomplete/incorrect responses unless the prompt is adjusted.

Workaround
Manually edit the system prompt to explicitly instruct the model to use the knowledge search tool first for document-based/factual questions (as documented in the Gen AI Playground RAG documentation). As a result, after updating the system prompt, RAG retrieval works and the model can answer based on the uploaded document content.

RHOAIENG-54005 - Generate MaaS Token Endpoint Removed - breaks Gen AI Studio Playground

The /v1/token API was removed and this endpoint was merged in with the new post creation of /v1/api-keys. As a result, Gen AI Playground cannot generate a token on the fly for MaaS and cannot talk to MaaS Models in 3.4 EA2.

Workaround
There is no existing workaround for this known issue. As a result, there is no access to MaaS and Playground in 3.4 EA2.

RHOAIENG-48753 - Pipeline Name must be DNS-compliant to use "Store pipeline definitions in Kubernetes"

Elyra does not convert the pipeline name to a DNS-compliant name when using the default Kubernetes storage. As a consequence, if you don’t use a DNS-compliant name when you start an Elyra pipeline, it gives a cryptic error "[TIP: did you mean to set https://ds-pipeline-dspa-robert-tests.apps.test.rhoai.rh-aiservices-bu.com/pipeline as the endpoint, take care not to include s at end]".

Workaround
Use DNS-compliant naming when running Elyra pipelines.

7.6. Issues discovered at version 3.4 EA1
Copy link

RHOAIENG-54101 - Deployments not listed in Model Registry on IBM Z

When you deploy a model from the Model Registry on IBM Z, the deployment does not appear under the Deployments tab in the Model Registry.

Workaround
Access and manage the deployment from the global Deployments page in the OpenShift AI dashboard.

RHOAIENG-53206 - Spark driver pods fail to communicate due to RpcTimeoutException

After installing the Spark Operator, Spark executor pods cannot communicate with the driver pod because the redhat-ods-applications namespace defaults to a "deny-all" traffic rule. SparkApplication pods hang and fail with an RpcTimeoutException.

Workaround

Create a NetworkPolicy in the redhat-ods-applications namespace to allow communication between the pods created by the SparkApplication controller: 

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: spark-operator-allow-internal
spec:
  podSelector:
    matchLabels:
      sparkoperator.k8s.io/launched-by-spark-operator: "true"
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - port: 7078
          protocol: TCP
        - port: 7079
          protocol: TCP
        - port: 4040
          protocol: TCP
      from:
        - podSelector: {}
        - namespaceSelector:
            matchLabels:
              network.openshift.io/policy-group: ingress

RHOAIENG-52130 - Workbenches with Feast integration fail to start due to missing ConfigMap

Workbenches with Feast integration enabled fail to start in OpenShift AI 3.4 EA1. Pods remain stuck in ContainerCreating state with the following error: 

[FailedMount] [Warning] MountVolume.SetUp failed for volume "odh-feast-config"
  configmap "jupyter-nb-kube-3aadmin-feast-config" not found
Workaround

Restart the Feast Operator after DSC deployment completes: 

$ kubectl rollout restart deployment/feast-operator-controller-manager -n redhat-ods-applications

RHOAIENG-53239 - Custom ServingRuntime required for IBM Z (s390x) vLLM Spyre deployments

When deploying models using the vLLM Spyre runtime on IBM Z (s390x) systems, the default ServingRuntime cannot be used directly for KServe-based deployments. Model deployment fails if the runtime is used without modification.

Workaround

Create a custom ServingRuntime by duplicating the vllm-spyre-s390x-runtime ServingRuntime and removing the command section from the container specification. Keep all other configuration, including environment variables, ports, and volume mounts, unchanged.

The following example shows only the affected section. Your complete ServingRuntime must include all other fields from the original template: 

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: vllm-spyre-s390x-runtime-copy
spec:
  containers:
    - name: kserve-container
      image: <image>
      # Remove the 'command' section that appears here in the original
      args:
        - --model=/mnt/models
        - --port=8000
        - --served-model-name={{.Name}}
      # ... keep all env, ports, volumeMounts from original ...
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top