Chapter 6. Resolved issues

This release provides security updates. For a complete list of updates, see the associated errata advisory on the Red Hat Customer Portal.

RHOAIENG-70232 - vLLM CPU model serving fails on IBM Z (s390x) during model warmup

Previously, on IBM Z, vLLM CPU model serving failed during model warmup due to a GCC version incompatibility. The vLLM container image included GCC 11.5.0, but PyTorch 2.11.0 required GCC 14 for JIT compilation features. As a consequence, the Torch Inductor C++ compilation step failed and the model did not load.

This issue is now resolved.

RHOAIENG-80354 - Perses observability dashboards fail to load in the OpenShift AI dashboard

Previously, after deploying OpenShift AI 3.5 with Cluster Observability Operator (COO) monitoring configured, the Perses proxy was not automatically set up. The Operator did not project the observability configuration from DataScienceClusterInitialization into the Dashboard custom resource (CR). As a consequence, navigating to Observe & Monitor in the OpenShift AI dashboard showed the error "Unable to reach observability dashboards" because frontend requests to the Perses API returned HTML instead of JSON.

This issue is now resolved.

RHOAIENG-65203 - Model Car (OCI) deployment fails for ONNX models with external data

Before this update, when you used the Model Car (OCI image) method to deploy an ONNX model split into model.onnx and model.onnx.data files, the MLServer runtime container could not access the external data file. The Model Car sidecar container exposed files by using cross-container symlinks instead of a shared volume, so the system could not load the model. Single-file model formats, such as SKLEARN, XGBoost, and LightGBM, were not affected.

This issue is now resolved.

RHOAIENG-62527 - RayJobs in the dashboard are missing a hyperlink to the Ray dashboard

Previously, when you created a RayJob, the Ray cluster name displayed in the OpenShift AI dashboard was plain text and did not include a hyperlink to the Ray cluster dashboard.

This issue is now resolved. The Ray cluster name under a RayJob is now a hyperlink to the Ray dashboard.

RHOAIENG-65143 - Models-as-a-Service UI does not detect custom gateway hostname

Previously, when the Models-as-a-Service (MaaS) gateway was configured with a custom hostname instead of the default maas.apps.<cluster_domain> pattern, the MaaS UI Backend-for-Frontend (BFF) service did not detect the custom hostname. The BFF service constructed the MaaS API URL by using the cluster’s external ingress domain, which failed when a custom gateway domain was in use or when the cluster was disconnected, making MaaS features unavailable in the OpenShift AI dashboard.

This issue is now resolved. Through fixes implemented in RHOAIENG-69335 and RHOAIENG-78980, the MaaS UI BFF service now fetches the gateway domain directly from the MaaS API rather than cluster ingress, and no longer crashes at startup when the MaaS API is not installed.

+ NOTE: If an administrator changes the gateway URL after the maas-ui has discovered a previous URL, you must restart the maas-ui pods to detect the new gateway URL.

RHOAIENG-74715 - DataScienceCluster status incorrectly reported aigateway module as not ready

Previously, when you enabled the aigateway component in the DataScienceCluster (DSC) custom resource, the DSC status might incorrectly report Ready: False with the message "Some modules are not ready: aigateway". This occurred because the status of the aigateway module did not propagate correctly, even when the AIGateway custom resource was fully provisioned and functional.

This issue is now resolved. The DSC status now correctly syncs with and reflects the Ready status of the aigateway component.

RHOAIENG-60855 - Upgrade error: OGX Operator produces invalid Deployment when storage is configured

Previously, when you upgraded OpenShift AI from 3.3 to 3.4, the Llama Stack Operator could fail to reconcile an existing LlamaStackDistribution custom resource that included persistent storage, for example, spec.server.storage.size: 2Gi. After a change that set the Deployment strategy to Recreate for ReadWriteOnce storage, the Operator left the previous rollingUpdate fields in place. Kubernetes rejected the Deployment because spec.strategy.type: Recreate and spec.strategy.rollingUpdate cannot be set together.

This issue no longer applies to the 3.3 to 3.4 LlamaStackDistribution upgrade path. OpenShift AI 3.5 uses the OGXServer custom resource, which creates a new Deployment instead of patching the LlamaStackDistribution Deployment that caused the conflict.

If you create an OGXServer without persistent storage and later add spec.workload.storage, the same error can still occur. This remaining case is fixed in OpenShift AI 3.6 EA1.

RHOAIENG-66859 - Evaluation jobs fail to complete with an MLflow experiment configured

Before this update, when you submitted an evaluation job with a configured MLflow experiment, the evaluation adapter failed after the evaluation completed successfully while trying to save results to MLflow. As a consequence, the evaluation remained indefinitely in a running state and did not report completion.

With this update, the issue is resolved.

RHOAIENG-59801 - Inconsistent resource limits when creating workloads with hardware profiles via API or CLI

Before this update, when you created a workbench or model deployment with a hardware profile using the API or CLI without pre-populating container resources, the operator webhook set the resource limit to the maxCount value defined in the hardware profile instead of the defaultCount value. This caused inconsistent resource allocation compared to workloads created through the dashboard.

This issue is resolved.

RHAIENG-3816 - Encrypted PDF uploads to Llama Stack vector stores fail on FIPS-enabled clusters

Before this update, on FIPS-enabled clusters, registering certain encrypted PDF files into Llama Stack vector stores failed because the underlying PDF parsing library used an MD5-based digest that is not allowed in FIPS mode.

This issue is resolved.

RHOAIENG-50523 - Unable to upload RAG documents in Gen AI Playground on disconnected clusters

Before this update, on disconnected clusters, uploading documents in the Gen AI Playground RAG section failed because Llama Stack attempted to download the ibm-granite/granite-embedding-125m-english embedding model from HuggingFace, even though the model was already included in the Llama Stack Distribution image.

This issue is resolved.

RHOAIENG-49017 - Upgrade RAGAS provider to Llama Stack 0.4.z / 0.5.z

Before this update, using the RAGAS evaluation provider required manually updating the Llama Stack distribution to use a specific workaround version of llama-stack-provider-ragas due to compatibility issues with newer Llama Stack versions.

This issue is resolved.

AIPCC-13675 - Feast and other python packages fail to start in workbench notebooks on IBM Z (s390x)

Before this update, when you start a workbench on IBM Z (s390x) systems, a segmentation fault (exit code 139) occurs during startup and the application stops. This issue occurs because the Protocol Buffers (protobuf) UPB C extension is not fully compatible with the s390x architecture. When you use this extension as the default protobuf implementation, the system crashes.

This issue is resolved.