Chapter 3. Technology Preview features

The MCP gateway Operator is an optional external prerequisite for OpenShift AI MCP management workflows, including agentic AI workflows, that route agent tool calls through a governed protocol gateway layer.

The MCP gateway Operator must be installed, configured, and upgraded independently from the OpenShift AI operator lifecycle. Cluster administrators are responsible for its full lifecycle management.

Generative AI model deployment workflows are in a unified dashboard experience, replacing separate entry points for different model serving runtimes with a single guided wizard. Previous versions of OpenShift AI required users to choose between separate UI workflows for different model serving runtimes: Legacy single-model serving, LLM-D distributed inference, and vLLM on Models-as-a-Service. This separation created confusion about which workflow to use for specific model types and required users to understand the underlying runtime differences before deploying models. There is a single Deploy model entry point for all generative AI models.

When you select Generative AI or Distributed inference with llm-d from the model framework list, the wizard guides you through the following:

The Model Context Protocol (MCP) Catalog in Red Hat OpenShift AI displays explicit support tier labels for server entries, allowing you to easily identify official support commitments before deployment. Support tiers are rendered directly in the MCP dashboard using accessible, visually distinct badges for the following categories:

You can run WideEP multi-rank inference with a single vllm serve command, enabling the Distributed Inference with llm-d Endpoint Picker to route requests to individual data-parallel (DP) ranks for prefix-cache-aware scheduling. The external multi-port DP mode provides one serving port per local rank for this routing. Additionally, the DP Supervisor starts these local ranks and exposes aggregated /health and /readyz endpoints on a separate admin port for Kubernetes probes, eliminating the need for custom shell wrappers. This routing method can deliver up to 26-30% higher throughput compared to hybrid load balancing.

Known limitations in this Technology Preview feature include: no partial-rank degradation if a single DP rank fails, no elastic EP scaling at runtime, and no support for gRPC transport.

For more information, see Launch a WideEP inference service with rank-aware routing.

Tool calling parameters pass through the full Distributed Inference with llm-d serving stack without modification. Tool definitions, tool choice settings, structured output schemas, and parallel_tool_calls in the request reach the model server unaltered through all layers of the serving stack, including the Gateway, Endpoint Picker, and the routing sidecar for disaggregated deployments. Additionally, tool call response objects and streaming tool call deltas are preserved on the return path.

This passthrough behavior applies to all Distributed Inference with llm-d deployment configurations, whether disaggregated or non-disaggregated. Non-tool-calling requests experience no impact on latency or throughput.

Platform administrators can provision isolated tenants for Models-as-a-Service by using a single custom resource. The controller automatically creates the tenant namespace, deploys the per-tenant API service, and configures authentication and rate-limit policies. The gateway and identity provider are external prerequisites that must exist before tenant creation. All controller-managed resources are cleaned up on deletion, with the namespace preserved to protect user data. Tenant users cannot discover or access models, API keys, or usage data belonging to other tenants.

Multi-tenancy supports long-lived API keys scoped to tenant identity, Bring Your Own Identity Provider configuration, per-tenant rate limits and quotas via subscriptions, and an access grant mechanism for exposing shared foundation models to specific tenants.

You can route inference requests through the Models-as-a-Service (MaaS) gateway using native provider API formats—such as the Anthropic Messages API at /v1/messages or the OpenAI Responses API at /v1/responses—without format translation. When the client API format matches the external model’s configured apiFormat, the gateway forwards the requests unchanged. This preserves provider-specific features, such as prompt caching and extended thinking. For direct Anthropic providers, beta flags are also preserved; however, for Vertex AI providers, beta headers and unsupported body fields are stripped. Additionally, AI coding tools can connect to a single MaaS gateway URL and switch between models mid-session using body-based model routing.

For more information, see Multi-provider API passthrough for external models.

You can enable cluster autoscaling for your Ray distributed workloads. Previously, network connectivity issues prevented the autoscaler sidecar from communicating with the Ray head pod due to missing TLS certificates. This update automatically injects the necessary client certificates into the sidecar container, allowing the underlying OpenShift worker pods in the Ray cluster to dynamically scale up or down based on workload demand. This eliminates the need for manual configuration edits and ensures efficient resource utilization.

This feature supports standard Ray cluster autoscaling; autoscaling with the Red Hat build of Kueue is not currently supported.

You can browse, load, and iterate on organization-curated prompts from global registry namespaces in the Gen AI Studio playground. Platform administrators choose shared MLflow prompt registry namespaces in the OdhDashboardConfig custom resource. Users with access can browse global prompts alongside their project prompts, with scope indicators showing each prompt’s source namespace.

When you change a global prompt, you can use Save As to create a personal copy in your local project namespace.

For more information about configuring global prompt registry namespaces, see Configure global prompt registry namespaces. For more information about using global prompts in the playground, see Reusable system instructions.

You can deploy and operate OpenCode, an open-source, terminal-based coding agent. OpenCode is the first coding agent validated to follow the onboarding pattern established by OpenClaw. This update confirms that agent platform operators, vLLM and OGX (formerly Llama Stack) inference backends, and MLflow tracing integration successfully generalize to coding-specific workloads.

This validation provides supported container images, operator-managed deployment manifests, verified inference backend interoperability, and MLflow tracing integration for running OpenCode on your OpenShift AI deployments.

You can export EvalHub operational metrics to OpenTelemetry Protocol (OTLP)-compatible observability backends using gRPC or HTTP transport. OTLP metrics export is available as a Technology Preview feature. To enable metrics export, set enableMetrics to true in the otel section of the EvalHub custom resource.

Metrics are exported at a configurable interval that you set in OTEL_METRIC_EXPORT_INTERVAL environment variable in the EvalHub custom resource spec.env field. Default value is 60 seconds.

If you enable both OTLP and Prometheus, EvalHub sends metrics to the configured OTLP backend and the Prometheus /metrics endpoint simultaneously.

You can export the following HTTP metrics in EvalHub:

Each metric includes Kubernetes resource attributes (service.name, k8s.namespace.name, k8s.pod.name, and k8s.node.name) to support multi-tenant filtering in your observability backend. When OTEL metrics export is not configured, only the existing Prometheus metrics are available, with no change to current behavior.

+ NOTE: You must deploy an OTLP-compatible collector, such as the OpenTelemetry Collector, in your cluster. EvalHub does not include a collector. The collector endpoint is shared between traces and metrics through the existing exporterEndpoint field in the otel configuration.

You can view registered external model endpoints and their associated provider details from the OpenShift AI dashboard. Navigate to AI hub Models and select the External models tab on the Model deployments page to see all ExternalModel resources in the selected project, including name, provider, and reconciliation status. You can expand each row to view associated ExternalProvider resources with connection details such as provider URL, authentication method, API format, and routing weight.

To enable this feature, set spec.dashboardConfig.externalModels to true in the OdhDashboardConfig custom resource. For more information, see About external models for Models-as-a-Service.

In OpenShift AI, you can access evaluation job execution logs directly through the EvalHub HTTP API and the evalhub CLI. This feature is available as a Technology Preview. By using the evalhub eval logs <job_id> command, you can inspect logs for jobs in any state, including running, completed, failed, or canceled, without requiring direct Kubernetes cluster access or tooling.

The API and EvalHub CLI support full log retrieval, log streaming for in-progress jobs, tailing the last specified number of lines, and formatted JSON output. This simplifies troubleshooting and enables automated CI/CD pipelines to retrieve failure details directly.

Models-as-a-Service (MaaS) includes a Loki-based structured log pipeline for showback data in addition to the existing metrics-based dashboard. The Loki-based pipeline provides 30-day retention on object storage to better support monthly billing cycles.

This enhancement also provides user-scoped, read-only dashboards with data isolation. Users, including administrators, can view only their own MaaS usage data, while administrators can continue to use the existing platform-wide dashboard for broader usage monitoring.

You can compare two or more completed evaluation runs side by side in the OpenShift AI dashboard. This Technology Preview feature enables you to select runs from the evaluations list, initiate a comparison, and view metrics and parameters for all selected runs in an embedded MLflow comparison view.

The comparison workflow supports the following capabilities:

The model catalog displays validated vLLM deployment arguments for models with confirmed tool-calling support. Models that have been validated for tool calling display a Validated Arguments section on the Model Details page, where you can expand the Tool Calling panel to view and copy the exact vllm serve CLI arguments required for tool calling, including --tool-call-parser, --reasoning-parser, --chat-template, and --enable-auto-tool-choice. You can also use the Validated Arguments filter to narrow the catalog to models with confirmed tool-calling support.

To enable this feature, set spec.dashboardConfig.toolCalling to true in the OdhDashboardConfig custom resource.

Red Hat OpenShift AI provides the docling-sdk-cuda-ubi9 container image for GPU-accelerated document conversion using the Docling SDK 2.88.0 with NVIDIA CUDA 13.0 support. The image includes all required dependencies for fully disconnected operation: PyTorch 2.12, Tesseract OCR, and pre-bundled machine learning models.

The following models are pre-bundled:

Red Hat OpenShift AI provides the docling-serve-cuda-ubi9 container image, which offers a REST API for on-demand document conversion, chunking, and GPU-accelerated parsing. The image is built as a thin layer on top of docling-sdk-cuda-ubi9, inheriting all machine learning models, dependencies, and CUDA GPU acceleration capabilities.

When you run the container, the docling-serve API server starts on port 5001. You can submit documents through HTTP requests and receive converted output in formats like Markdown or Docling Doc JSON. This image enables real-time document processing workflows with GPU acceleration for applications that require on-demand conversion rather than batch processing.

You can save, version, and reuse system instructions as named prompts in Gen AI Studio. Prompts are stored in the MLflow prompt registry and scoped to your project, so any team member can browse and load them.

Prompts can have {{variable}} placeholders that are filled with specific values before inference. When you load a prompt that has placeholders, the playground displays an input panel where you enter values.

You can route EvalHub evaluation jobs through Red Hat build of Kueue LocalQueues by specifying a queue name when creating an evaluation job. This integration provides fair resource sharing, quota management, priority-based scheduling, and queue-based admission control across evaluation workloads.

If Red Hat build of Kueue is not installed on your cluster, evaluation jobs use the default Kubernetes scheduler without requiring changes to your job submission logic.

For more information, see EvalHub Kueue integration overview.

Red Hat OpenShift AI 3.5 EA1 updates the guardrails experience in the Gen AI Studio playground. Guardrails are powered by NeMo Guardrails, providing more reliable and consistent safety checks for model interactions.

You can independently control guardrails for user input and model output directly from the playground settings:

The OpenAI Conversations API is available on OGX. This development tool allows you to build context-aware, multi-turn AI applications.

Key capabilities of the Conversations API include:

You can pre-download and cache large language model (LLM) artifacts on node-local Non-Volatile Memory Express (NVMe) storage to reduce InferenceService cold-start latency. When a model serving pod starts on a node with a cached model copy, the pod mounts the local copy instead of downloading from remote storage, which makes autoscaling practical for latency-sensitive workloads.

A cluster administrator enables model caching through the DataScienceCluster custom resource and configures target nodes and storage capacity. Model Cache supports both cluster-scoped caching, where any InferenceService in any namespace can use the cached model, and namespace-scoped caching for namespace-level isolation. For more information, see Model cache for faster inference startup.

You can discover and select an existing Kubernetes Gateway resource from the model serving UI when deploying LLMInferenceService models. This Technology Preview feature enables self-service Gateway management, supports multitenant namespace-scoped network isolation, and provides programmatic access through the Gateway discovery REST API.

Gateway discovery and creation is disabled by default and must be explicitly enabled by a cluster administrator through the dashboard configuration.

You can deploy models using the vLLM runtime through Models-as-a-Service (MaaS). This Technology Preview feature enables you to serve large language models with vLLM’s high-performance inference capabilities while benefiting from MaaS governance and subscription-based controls.

Key capabilities include:

As a cluster administrator, you can monitor platform-wide Models-as-a-Service (MaaS) usage through a dedicated observability dashboard embedded in the OpenShift AI console. This Technology Preview feature provides comprehensive usage metrics for cost attribution and showback reporting to finance teams.

Key capabilities include:

You can route model inference requests to external model providers through the Models-as-a-Service (MaaS) inference gateway. This Technology Preview feature enables you to apply MaaS governance policies, token tracking, and rate limiting to third-party LLM services such as OpenAI, Anthropic, or other external providers.

Key capabilities include:

Red Hat OpenShift AI provides recommended vLLM runtime configurations for select high-demand models. These configurations are designed to help users achieve maximum performance on specific hardware profiles, matching or exceeding industry benchmarks for low-latency and high-throughput serving.

Previously, users had to manually tune vLLM arguments to find the sweet spot for performance. Validated YAML templates are embedded directly in the model card Readme section as markdown, serving as a reference for manual configuration.

Key features of this update include:

Model registry provides cryptographic signing and verification of AI artifacts by using the Python client API. This Technology Preview feature enables you to sign artifacts to establish authenticity and verify signatures to confirm integrity. Artifact signing provides foundational capabilities for model serving verification and AI asset provenance tracking in future releases.

For more information, see the Kubeflow Model Registry Python client documentation and Kubeflow async upload job documentation.

Red Hat OpenShift AI includes a Technology Preview of the Evaluation Stack user interface. The Evaluation Stack UI provides a guided workflow in the OpenShift AI dashboard for running model evaluations without requiring command-line tools or notebook workflows.

This Technology Preview introduces the following capabilities:

Red Hat OpenShift AI includes a dedicated suite of text embedding models within the Model Catalog that can be deployed on vLLM. This Technology Preview feature allows data scientists and AI engineers to discover and deploy models designed specifically for vector generation, a critical component for Retrieval-Augmented Generation (RAG) and semantic search workflows. By separating these models from standard generative large language models (LLMs), OpenShift AI provides a streamlined experience for users building vector-generation engines.

Key features of this update include:

To view embedding models in the Catalog, navigate to AI hub Models Catalog in the OpenShift AI dashboard, select the Other models category, and look for the text-embedding tag on model cards. You can deploy a model directly from its card by following the deployment wizard to configure the serving runtime and hardware profile. Once deployed, the model provides an API endpoint that generates numerical embeddings for text inputs for use in downstream vector databases or RAG pipelines.

+ NOTE: Some models, such as Nomic Embed Text v1.5, require extra vLLM parameters to be set, such as --trust-remote-code. Refer to upstream Hugging Face model cards for specific details.

The model deployment wizard includes a YAML viewer that provides real-time YAML preview and manual editing capabilities for LLMInferenceService deployments. You can toggle between Form and YAML views to see the generated LLMInferenceService YAML as you fill out the deployment form. The preview updates automatically as form fields change, enabling you to verify configuration before deployment. You can also enter Manual Edit Mode to edit YAML directly for advanced Distributed Inference with llm-d parameters not exposed in the form, such as autoscaling, disaggregated serving, LoRA adapters, and custom runtime configurations.

This Technology Preview feature introduces the following capabilities:

You can compare results across multiple configurations in the Playground by using multiple chat panes side-by-side. Each pane supports unique configurations for models, prompts, MCP servers, guardrails, and knowledge sources. You can run synchronized prompts across all panes or use independent prompts for each pane.

Key capabilities include:

The Gen AI Playground provides access to basic safety guardrails from OGX. You can enable or disable guardrails in the playground interface to filter unsafe content and detect prompt injection attempts.

The following guardrails are available:

Starting with OpenShift AI 3.3, OGX provides OpenAI-compatible grounding and citation annotations for search-backed responses as a Technology Preview feature.

This enhancement enables retrieval-augmented generation (RAG) applications to trace generated responses back to source documents by using the same annotation schemas returned by OpenAI Search and Responses APIs. The feature supports document source attribution and preserves citation metadata in API responses, allowing existing OpenAI client applications to consume citation information without code changes.

This capability improves transparency, auditability, and explainability for enterprise RAG workloads, and serves as a foundation for future advanced tracing and observability features in OGX. For more information, see OpenAI-compatible file citation annotations.

The OGX Operator in OpenShift AI 3.3 offers ConfigMap driven image updates for OGXServer resources. This allows you to patch security or bug fixes without new Operator versions. To enable this feature, update your ConfigMap with the following parameters:

 image-overrides: |
    starter-gpu: registry.redhat.io/rhoai/odh-ogx-core-rhel9:v3.3
    starter: registry.redhat.io/rhoai/odh-ogx-core-rhel9:v3.3

Using the starter-gpu and starter distributions names as the key allows the operator to apply these overrides automatically.

To update the OGX Distributions image for all starter distributions, run the following command:

$ kubectl patch configmap ogx-operator-config -n ogx-k8s-operator-system --type merge -p '{"data":{"image-overrides":"starter: quay.io/opendatahub/ogx:latest"}}'

This allows the OGXServer resources to restart with the new image.

Starting with OpenShift AI 3.2, you can use PostgreSQL with the pgvector extension as a remote vector store provider for the OGX vector_store endpoint as a Technology Preview feature.

This enhancement enables vector storage backed by PostgreSQL, providing durable and transactional persistence for vector embeddings. For more information, see OGX API provider support and Deploying a PostgreSQL instance with pgvector.

OpenShift AI 3.2 allows you to customize your embedding models as a Technology Preview feature. In the version of OGX shipped in OpenShift AI 3.2, vLLM controls embeddings by default. You can update the VLLM_EMBEDDING_URL environment variable in your OGXServer custom resource to enable embeddings, or you can use custom embeddings providers. For example:

  - name: ENABLE_SENTENCE_TRANSFORMERS
    value: "true"
  - name: EMBEDDING_PROVIDER
    value: "sentence-transformers"

You can use the Guardrails Orchestrator from TrustyAI with OGX as a Technology Preview feature.

This integration enables built-in detection and evaluation workflows to support AI safety and content moderation. When TrustyAI is enabled and the FMS Orchestrator and detectors are configured, no manual setup is required.

To activate this feature, set the following field in the DataScienceCluster custom resource for the OpenShift AI Operator: spec.ogx.managementState: Managed

For more information, see the TrustyAI FMS Provider on GitHub: TrustyAI FMS Provider.

A new AI Available Assets page enables AI engineers and application developers to view and consume deployed AI resources within their projects.

This enhancement introduces a filterable UI that lists available models and Model Context Protocol (MCP) servers in the selected project, allowing users with appropriate permissions to identify accessible endpoints and integrate them directly into the AI Playground or other applications.

The Generative AI (GenAI) Playground introduces a unified, interactive experience within the OpenShift AI dashboard for experimenting with foundation and custom models.

Users can test prompts, compare models, and evaluate Retrieval-Augmented Generation (RAG) workflows by uploading documents and chatting with their content. The GenAI Playground also supports integration with approved Model Context Protocol (MCP) servers and enables export of prompts and agent configurations as runnable code for continued iteration in local IDEs.

Chat context is preserved within each session, providing a suitable environment for prompt engineering and model experimentation.

You can install and operate OGX and RAG/Agentic components in fully disconnected (air-gapped) OpenShift AI environments.

This enhancement enables secure deployment of OGX features without internet access, allowing organizations to use AI capabilities while maintaining compliance with strict network security policies.

This feature is available as a Technology Preview.

The Feature Store is integrated with OpenShift AI, data science projects, and workbenches. This integration also introduces centrally managed, role-based access control (RBAC) capabilities for improved governance.

These enhancements provide two key capabilities:

The Feature Store component includes a web-based user interface (UI).

You can use the UI to view registered Feature Store objects and their relationships, such as features, data sources, entities, and feature services.

To enable the UI, edit your FeatureStore custom resource (CR) instance. When you save the change, the Feature Store Operator starts the UI container and creates an OpenShift route for access.

For more information, see Setting up the Feature Store user interface for initial use.

With this release, the OGX Technology Preview feature enables Retrieval-Augmented Generation (RAG) and agentic workflows for building next-generation generative AI applications. It supports remote inference, built-in embeddings, and vector database operations. It also integrates with providers like TrustyAI’s provider for safety and Trusty AI’s LM-Eval provider for evaluation.

This preview includes tools, components, and guidance for enabling the OGX Operator, interacting with the RAG Tool, and automating PDF ingestion and keyword search capabilities to enhance document discovery.

Centralized platform observability, including metrics, traces, and built-in alerts, is available as a Technology Preview feature. This solution introduces a dedicated, pre-configured observability stack for OpenShift AI that allows cluster administrators to perform the following actions:

The OGX Distribution includes version 0.3.0 as a Technology Preview feature.

This update introduces several enhancements, including expanded support for retrieval-augmented generation (RAG) pipelines, improved evaluation provider integration, and updated APIs for agent and vector store management. It also provides compatibility updates aligned with recent OpenAI API extensions and infrastructure optimizations for distributed inference.

The previously supported version was 0.2.22.

OpenShift AI supports Kubernetes Event-driven Autoscaling (KEDA) in its KServe RawDeployment mode. This Technology Preview feature enables metrics-based autoscaling for inference services, allowing for more efficient management of accelerator resources, reduced operational costs, and improved performance for your inference services.

To set up autoscaling for your inference service in KServe RawDeployment mode, you need to install and configure the OpenShift Custom Metrics Autoscaler (CMA), which is based on KEDA.

For more information about this feature, see: Configuring metrics-based autoscaling.

You can create and manage Ray Jobs on Ray Clusters directly through the CodeFlare SDK.

This enhancement aligns the CodeFlare SDK workflow with the KubernetesFlow Training Operator (KFTO) model, where a job is created, run, and completed automatically. This enhancement simplifies manual cluster management by preventing Ray Clusters from remaining active after job completion.

You can use a custom flow estimator for synthetic data generation (SDG) pipelines.

For supported and compatible tagged SDG teacher models, the estimator helps you evaluate a chosen teacher model, custom flow, and supported hardware on a sample dataset before running full workloads.

OGX core can deploy and run efficiently on single node OpenShift (SNO).

This enhancement optimizes component startup and resource usage so that OGX can operate reliably in single-node cluster environments.

You can use the FAISS (Facebook AI Similarity Search) library as an inline vector store in OpenShift AI.

FAISS is an open-source framework for high-performance vector search and clustering, optimized for dense numerical embeddings with both CPU and GPU support. When enabled with an embedded SQLite backend in the OGX Distribution, FAISS stores embeddings locally within the container, removing the need for an external vector database service.

You can install and manage Feature Store as a configurable component in OpenShift AI. Based on the open-source Feast project, Feature Store acts as a bridge between ML models and data, enabling consistent and scalable feature management across the ML lifecycle.

This Technology Preview release introduces the following capabilities:

You can deploy OGX and RAG or agentic solutions in regulated environments that require FIPS compliance.

This enhancement provides FIPS-certified and compatible deployment patterns to help organizations meet strict regulatory and certification requirements for AI workloads.

Validated sdg_hub example notebooks are available to provide a notebook-driven user experience in OpenShift AI 3.0.

These notebooks support multiple Red Hat platforms and enable customization through SDG pipelines. They include examples for the following use cases:

You can use the Retrieval-Augmented Generation Assessment (RAGAS) evaluation provider to measure the quality and reliability of RAG systems in OpenShift AI.

RAGAS provides metrics for retrieval quality, answer relevance, and factual consistency, helping you identify issues and optimize RAG pipeline configurations.

The integration with the OGX evaluation API supports two deployment modes:

The hardware profiles feature enables users to target specific worker nodes for workbenches or model-serving workloads. It allows users to target specific accelerator types or CPU-only nodes.

This feature replaces the current accelerator profiles feature and container size selector field, offering a broader set of capabilities for targeting different hardware configurations. While accelerator profiles, taints, and tolerations provide some capabilities for matching workloads to hardware, they do not ensure that workloads land on specific nodes, especially if some nodes lack the appropriate taints.

The hardware profiles feature supports both accelerator and CPU-only configurations, along with node selectors, to enhance targeting capabilities for specific worker nodes. Administrators can configure hardware profiles in the settings menu. Users can select the enabled profiles using the UI for workbenches, model serving, and AI pipelines where applicable.