OpenShift Container Platform 4.9 release notes

Nodes installed with the coreos-installer program previously retained the installation Ignition config in the /boot/ignition/config.ign file. Starting with the OpenShift Container Platform 4.9 installation image, that file is removed when the node is provisioned. This change does not affect clusters that were installed on previous OpenShift Container Platform versions because they still use an older bootimage.

RHCOS now uses Red Hat Enterprise Linux (RHEL) 8.4 packages in OpenShift Container Platform 4.9. These packages provide you the latest fixes, features, and enhancements, such as NetworkManager features, as well as the latest hardware support and driver updates.

OpenShift Container Platform 4.9 introduces support for installing a cluster on Azure Stack Hub using user-provisioned infrastructure.

You can incorporate example Azure Resource Manager (ARM) templates provided by Red Hat to assist in the deployment process, or create your own. You are also free to create the required resources through other methods; the ARM templates are just an example.

See Installing a cluster on Azure Stack Hub using ARM templates for details.

During the upgrade process, nodes in the cluster might become temporarily unavailable. In the case of worker nodes, the machine health check might identify such nodes as unhealthy and reboot them. To avoid rebooting such nodes, OpenShift Container Platform 4.9 introduces the cluster.x-k8s.io/paused="" annotation to let you pause the MachineHealthCheck resources before updating the cluster.

For more information, see Pausing a MachineHealthCheck resource.

The OpenShift Container Platform installation program for Microsoft Azure now creates subnets as large as possible within the machine CIDR. This lets the cluster use a machine CIDR that is appropriately sized to accommodate the number of nodes in the cluster.

OpenShift Container Platform 4.9 introduces support for AWS regions in China. You can now install and update OpenShift Container Platform clusters in the cn-north-1 (Beijing) and cn-northwest-1(Ningxia) regions.

For more information, see Installing a cluster on AWS China.

In OpenShift Container Platform 4.9, you can expand an installer provisioned cluster deployed using the provisioning network by using Virtual Media on the baremetal network. You can use this feature when the ProvisioningNetwork configuration setting is set to Managed. To use this feature, you must set the virtualMediaViaExternalNetwork configuration setting to true in the provisioning custom resource (CR). You must also edit the machine set to use the API VIP address. See Preparing to deploy with Virtual Media on the baremetal network for details.

OpenShift Container Platform 4.9 uses Kubernetes 1.22, which removed a significant number of deprecated v1beta1 APIs.

OpenShift Container Platform 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.8 to 4.9. This is to help prevent issues after upgrading to OpenShift Container Platform 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All OpenShift Container Platform 4.8 clusters require this administrator acknowledgment before they can be upgraded to OpenShift Container Platform 4.9.

For more information, see Preparing to update to OpenShift Container Platform 4.9.

OpenShift Container Platform 4.9 introduces support for installation on Red Hat OpenStack Platform (RHOSP) deployments that rely on PCI passthrough.

OpenShift Container Platform 4.9 supports etcd 3.5. Before you upgrade the cluster, verify that a valid etcd backup exists. An etcd backup ensures that the cluster can be restored if an upgrade failure occurs. In OpenShift Container Platform 4.9, etcd upgrades are automatic. Depending on the cluster’s transition state to version 4.9, an etcd backup might be available. However, verifying that a backup exists before the cluster upgrade starts is recommended.

OpenShift Container Platform 4.9 introduces support for installing a cluster on IBM Cloud® using installer-provisioned infrastructure. The procedure is nearly identical to installer-provisioned infrastructure on bare metal with these differences:

See Deploying installer-provisioned clusters on IBM Cloud for details.

OpenShift Container Platform 4.9 adds BIOS configuration support for worker nodes when deploying installer-provisioned clusters on Fujitsu hardware and using the Fujitsu integrated Remote Management Controller (iRMC). See Configuring BIOS for worker node for details.

With this update, administrators now have the ability to access node logs from the Node page. To review the node logs, you can switch between individual log files and journal log units by clicking the Logs tab.

You now have the ability to filter by node type in the Cluster utilization card on the cluster dashboard. Additional node types will appear in the list when created.

This update adds a User Preferences page for customizing settings, such as default project, perspective, and topology view.

With this update, you can hide default projects from the Projects dropdown in the web console masthead. You can still toggle to show default projects before you search and filter.

With this update, you can now add user preferences in the web console. Users can select their default perspective, project, topology, and other preferences.

You now have more fine-grained control over the audit logging level for OpenShift Container Platform. You can use custom rules to specify a different audit policy profile (Default, WriteRequestBodies, AllRequestBodies, or None) for different groups.

For more information, see Configuring the audit log policy with custom rules.

You can now disable audit logging for OpenShift Container Platform by using the None audit policy profile.

For more information, see Disabling audit logging.

You can now customize the URL for the internal OAuth server. For more information, see Customizing the internal OAuth server URL.

OpenShift Container Platform 4.9 provides new procedures for ongoing maintenance of NBDE-configured systems. NBDE allows you to encrypt root volumes of hard drives on physical and virtual machines without having to manually enter a password when restarting machines. For more information, see About disk encryption technology.

In OpenShift Container Platform 4.9, etcd certificates are automatically rotated and are managed by the system.

The Kubernetes API server TLS security profile setting is now also honored by etcd.

OpenShift Container Platform 4.9 introduces the following updates to PTP:

For more information, see Configuring linuxptp services as boundary clock.

Fast event notifications for PTP events are now available for bare-metal clusters. The PTP Operator generates event notifications for every configured PTP-capable network interface. Events are made available through a REST API for applications running on the same node. Fast event notifications are transported by an Advanced Message Queuing Protocol (AMQP) message bus provided by the AMQ Interconnect Operator.

For more information, see About PTP and clock synchronization error events.

The egress IP feature of OVN-Kubernetes now balances network traffic approximately equally across nodes for a given namespace, if that namespace is assigned multiple egress IP addresses. Each IP address must reside on a different node. For more information, refer to Configuring egress IPs for a project for OVN-Kubernetes.

The containerized Data Plane Development Kit (DPDK) is now GA in OpenShift Container Platform 4.9. For more information, see Using virtual functions (VFs) with DPDK and RDMA modes.

SR-IOV now supports vhost-net for use with Fast Datapath DPDK applications on Intel and Mellanox NICs. You can enable this feature by configuring the SriovNetworkNodePolicy resource. For more information, see SR-IOV network node configuration object.

Single node clusters support SR-IOV hardware and the SR-IOV Network Operator. Be aware that configuring an SR-IOV network device causes the single node to reboot and that you must configure the disableDrain field for the Operator. For more information, see Configuring the SR-IOV Network Operator.

OpenShift Container Platform 4.9 adds support for additional Broadcom and Intel hardware.

For more information, see the supported devices.

This release introduces the MetalLB Operator. After installing and configuring the MetalLB Operator, you can deploy MetalLB to provide a native load balancer implementation for services on bare-metal clusters. Other on-premise infrastructures that are like bare metal can also benefit.

The Operator introduces a custom resource, AddressPool. You configure address pools with ranges of IP addresses that MetalLB can assign to services. When you add a service of type LoadBalancer, MetalLB assigns an IP address from a pool.

For this release, Red Hat only supports using MetalLB in layer 2 mode.

For more information, see About MetalLB and the MetalLB Operator.

The CNI VRF plugin was previously introduced as a Technology Preview feature in OpenShift Container Platform 4.7 and is now generally available in OpenShift Container Platform 4.9.

For more information, see Assigning a secondary network to a VRF.

This release introduces six timeout configurations for the Ingress Controller tuningOptions parameter:

For more information, see Ingress controller configuration parameters.

You can now configure the Ingress Controller to enable mutual TLS (mTLS) authentication by setting spec.clientTLS. The clientTLS field specifies configuration for the Ingress Controller to verify client certificates.

For more information, see Configuring Mutual TLS Authentication.

Cluster administrators can specify a custom HTTP error code response page for either 503, 404, or both error pages.

For more information, see Customizing HAProxy error code response pages.

In OpenShift Container Platform 4.9, the provisioningNetworkInterface configuration setting for installer-provisioned clusters is optional. The provisioningNetworkInterface configuration setting identifies the NIC name used for the provisioning network. In OpenShift Container Platform 4.9, you can alternatively specify the bootMACAddress configuration setting in the install-config.yml file, which enables Ironic to identify the IP address for the NIC connected to the provisioning network and bind to it. You can also omit the provisioningInterface configuration setting in the provisioning custom resource so that the provisioning custom resource uses the bootMACAddress configuration setting instead.

In OpenShift Container Platform 4.9, you can now change the DNS Operator managementState. The managementState of the DNS Operator is set to Managed by default, which means that the DNS Operator is actively managing its resources. You can change it to Unmanaged, which means the DNS Operator is not managing its resources.

The following are use cases for changing the DNS Operator managementState:

For more information, see Changing the DNS Operator managementState.

For clusters that run on RHOSP, you can now configure Octavia for load balancing as a cloud provider option.

For more information, see Setting cloud provider options.

This release adds Ingress Controller support for TLS 1.3 and the Modern profile in HAProxy.

For more information, see Ingress Controller TLS security profiles.

Cluster administrators can configure HTTP Strict Transport Security (HSTS) verification on a per-domain basis with the addition of an admission plugin for the router, called route.openshift.io/RequiredRouteAnnotations. If a cluster administrator configures this plugin to enforce HSTS, then any newly created route must be configured with a compliant HSTS Policy, which is verified against the global setting on the cluster Ingress configuration, called ingresses.config.openshift.io/cluster.

For more information, see HTTP Strict Transport Security.

In OpenShift Container Platform 4.9 you can now configure the Ingress Controller to log or ignore empty requests by setting the logEmptyRequests and HTTPEmptyRequestsPolicy fields.

For more information, see Ingress controller configuration parameters.

Logging in to the web console with the cluster-admin role now enables you to create new network policies in any namespace in the cluster from a form in the console. Previously, this could only be done directly in YAML.

OpenShift Container Platform is capable of provisioning persistent volumes (PVs) using the Container Storage Interface (CSI) driver for AWS Elastic Block Store (EBS). This feature was previously introduced as a Technology Preview feature in OpenShift Container Platform 4.5 and is now generally available and enabled by default in OpenShift Container Platform 4.9.

For more information, see AWS EBS CSI Driver Operator.

OpenShift Container Platform is capable of provisioning PVs using the CSI driver for Azure Stack Hub Storage. Azure Stack Hub, which is part of the Azure Stack portfolio, allows you to run apps in an on-premises environment and deliver Azure services in your datacenter. The Azure Stack Hub CSI Driver Operator that manages this driver is new for 4.9 and generally available.

For more information, see Azure Stack Hub CSI Driver Operator.

OpenShift Container Platform is capable of provisioning PVs using the CSI driver for AWS Elastic File Service (EFS). The AWS EFS CSI Driver Operator that manages this driver is in Technology Preview.

For more information, see AWS EFS CSI Driver Operator.

Starting with OpenShift Container Platform 4.8, automatic migration for in-tree volume plugins to their equivalent CSI drivers became available as a Technology Preview feature. This feature now supports automatic migration from Google Compute Engine Persistent Disk (GCE PD) in-tree plugin to the Google Cloud Platform (GCP) Persistent Disk CSI driver.

For more information, see CSI Automatic Migration.

Starting with OpenShift Container Platform 4.8, automatic migration for in-tree volume plugins to their equivalent CSI drivers became available as a Technology Preview feature. This feature now supports automatic migration from the Azure Disk in-tree plugin to the Azure Disk CSI driver.

For more information, see CSI Automatic Migration.

The vSphere CSI Operator Driver storage class now uses vSphere’s storage policy. OpenShift Container Platform automatically creates a storage policy that targets datastore configured in cloud configuration.

For more information, see VMWare vSphere CSI Driver Operator.

OpenShift Container Platform 4.9 provides the following new metrics for the Local Storage Operator:

For more information, see Persistent storage using local volumes.

OpenShift Container Platform 4.9 adds resizing capability to the oVirt CSI Driver, which allows users to increase the size of their existing persistent volume claims (PVCs). Prior to this feature, users had to create new PVCs with the increased size, and move all of the content from the old persistent volume (PV) to the new PV, which could result in data loss. Now, users can edit the existing PVC and the oVirt CSI Driver will resize the underlying oVirt disk.

In OpenShift Container Platform 4.9, the integrated Image Registry uses Azure Blob Storage for clusters installed on Microsoft Azure Stack Hub using user-provisioned infrastructure.

See Installing a cluster on Azure Stack Hub using ARM templates for details.

Starting in OpenShift Container Platform 4.9, Operator Lifecycle Manager (OLM) supports Kubernetes 1.22. As a result, a significant number of v1beta1 APIs have been removed and updated to v1. Operators that depend on the removed v1beta1 APIs will not run on OpenShift Container Platform 4.9. Cluster administrators should upgrade their installed Operators to the latest channel before upgrading a cluster to OpenShift Container Platform 4.9.

File-based catalogs are the latest iteration of the catalog format in Operator Lifecycle Manager (OLM). The format is a plain text-based (JSON or YAML) and declarative config evolution of the earlier, and now deprecated, SQLite database format, and it is fully backwards compatible. The goal of this format is to enable Operator catalog editing, composability, and extensibility.

For more information about the file-based catalog specification, see Operator Framework packaging format.

For instructions about creating file-based catalogs by using the opm CLI, see Managing custom catalogs.

Operator Lifecycle Manager (OLM) is now available on Single Node OpenShift (SNO) clusters, enabling self-service Operator installations.

Because administrators should not require an understanding of the interaction process between the various low-level APIs or access to the Operator Lifecycle Manager (OLM) pod logs to successfully debug such issues, OpenShift Container Platform 4.9 introduces the following enhancements in OLM to provide administrators with more comprehensible error reporting and messages:

To avoid cluster upgrades potentially leaving Operator installations in an unsupported state or without a continued update path, you can enable automatically changing your Operator catalog’s index image version as part of cluster upgrades.

Set the olm.catalogImageTemplate annotation to your catalog image name and use one or more of the Kubernetes cluster version variables when constructing the template for the image tag.

For more information, see Image template for custom catalog sources.

An OpenShift Container Platform cluster can be configured in high-availability (HA) mode, which uses multiple nodes, or in non-HA mode, which uses a single node. A single node cluster, also known as Single Node OpenShift (SNO), is likely to have more conservative resource constraints. Therefore, it is important that Operators installed on a single node cluster can adjust accordingly and still run well.

By accessing the cluster high-availability mode API provided in OpenShift Container Platform, Operator authors can use the Operator SDK to enable their Operator to detect a cluster’s infrastructure topology, either HA or non-HA mode. Custom Operator logic can be developed that uses the detected cluster topology to automatically switch the resource requirements, both for the Operator and for any Operands or workloads it manages, to a profile that best fits the topology.

For more information, see High-availability or single node cluster detection and support.

Operator authors can now develop Operators that support network proxies. Operators with proxy support inspect the Operator deployment for environment variables and pass the variables on to the required Operands. Cluster administrators configure proxy support for the environment variables that are handled by Operator Lifecycle Manager (OLM). For more information, see the Operator SDK tutorials for developing Operators using Go, Ansible, and Helm.

You can now check bundle manifests for APIs removed from Kubernetes 1.22 by using the Operator Framework suite of tests with the bundle validate subcommand.

For example:

If your bundle manifest includes APIs removed from Kubernetes 1.22, the command displays a warning message. The warning message indicates which APIs you need to migrate and links to the Kubernetes API migration guide.

See the table of beta APIs removed from Kubernetes 1.22 and the Operator SDK CLI reference for more information.

This release introduces support for using wildcard domains as registry sources in your image registry settings. With a wildcard domain, such as *.example.com, you can set your cluster to push and pull images from multiple subdomains without having to manually enter each one. For more information, see Image controller configuration parameters.

Starting in OpenShift Container Platform 4.9, you can now use Red Hat Enterprise Linux (RHEL) 8.4 for compute machines. Previously, RHEL 8 was not supported for compute machines.

You cannot upgrade RHEL 7 compute machines to RHEL 8. You must deploy new RHEL 8 hosts, and the old RHEL 7 hosts should be removed.

Scheduling pods using a scheduler profile is now generally available. This is a replacement for configuring a scheduler policy. The following scheduler profiles are available:

For more information, see Scheduling pods using a scheduler profile.

The following descheduler profiles are now available:

You can also customize the pod lifetime value for the LifecycleAndUtilization profile.

For more information, see Evicting pods using the descheduler.

When configuring the docker/config.json file to allow pods to pull images from private registries, you can now list specific repositories in the same registry, each with credentials specific to that registry path. Previously, you could list only one repository from a given registry. You can also now define a registry with a specific namespace.

Node-related metrics and alerts have been enhanced to give you an earlier indication of when the stability of a node is compromised.

The Poison Pill Operator now provides enhanced remediation with the use of watchdog devices. For more information, see About watchdog devices.

You can use the Node Health Check Operator to deploy the NodeHealthCheck controller. The controller identifies unhealthy nodes and uses the Poison Pill Operator to remediate the unhealthy nodes.

Updates to versions of monitoring stack components and dependencies include the following:

The link to the third-party Prometheus UI is removed from the Observe → Metrics page in the OpenShift Container Platform web console. You can still access the route to the Prometheus UI in the web console in the Administrator perspective by navigating to the Networking → Routes page in the openshift-monitoring project.

Because running the default Grafana dashboard can take resources from user workloads, you can disable the Grafana dashboard deployment.

You can now use the Special Resource Operator (SRO) to help manage the deployment of kernel modules and drivers on an existing OpenShift Container Platform cluster. This is currently a Technology Preview feature.

For more information, see About the Special Resource Operator.

The Memory Manager, a kubelet subcomponent configured by the Performance Addon Operator, is now enabled by default for all pods running on the node that is configured with one of the following Topology Manager policies:

For more information, see Topology Manager policies.

OpenShift Container Platform 4.9 introduces two additional tools to measure system latency:

For more information, see Running the latency tests.

Updated guidance around cluster maximums for OpenShift Container Platform 4.9 is now available.

Use the OpenShift Container Platform Limit Calculator to estimate cluster limits for your environment.

OpenShift Container Platform 4.9 supports zero touch provisioning (ZTP), which allows you to provision new edge sites with declarative configurations of bare metal equipment at remote sites. ZTP uses the GitOps deployment set of practices for infrastructure deployment. GitOps achieves these tasks using declarative specifications stored in Git repositories, such as YAML files and other defined patterns, to provide a framework for deploying the infrastructure. The declarative output is leveraged by the Open Cluster Manager (OCM) for multisite deployment. For more information, see Provisioning edge sites at scale.

In OpenShift Container Platform 4.9, Insights Operator can import RHEL Simple Content Access (SCA) certificates from Red Hat OpenShift Cluster Manager.

For more information, see Importing RHEL Simple Content Access certificates with Insights Operator.

In OpenShift Container Platform 4.9, the Insights Operator collects the following additional information:

With this additional information, Red Hat can provide improved remediation steps in Insights Advisor.

With this release, installations on Microsoft Azure Stack Hub can be performed by configuring the Cloud Credential Operator (CCO) in manual mode.

For more information, see Using manual mode.

The SQLite database format used by Operator Lifecycle Manager (OLM) for catalogs and index images has been deprecated, including the related opm CLI commands. Cluster administrators and catalog maintainers are encouraged to familiarize themselves with the new file-based catalog format introduced in OpenShift Container Platform 4.9 and begin migrating catalog workflows.

Installing a cluster on VMware vSphere version 6.7 Update 2 or earlier and virtual hardware version 13 is now deprecated. Support for these versions will end in a future version of OpenShift Container Platform.

Hardware version 15 is now the default for vSphere virtual machines in OpenShift Container Platform. Hardware version 15 will be the only supported version in a future version of OpenShift Container Platform.

The instance_type_id installation configuration parameter is deprecated and will be removed in a future release.

This release removes the OpenShift Container Platform Metering Operator feature.

Kubernetes 1.22 removed the following deprecated v1beta1 APIs. Migrate manifests and API clients to use the v1 API version. For more information about migrating removed APIs, see the Kubernetes documentation.

The deprecated v1beta1 API for the descheduler has been removed in OpenShift Container Platform 4.9. Migrate any resources using the descheduler v1beta1 API version to v1.

The deprecated dhclient binary has been removed from RHCOS. Starting with OpenShift Container Platform 4.6, RHCOS switched to using NetworkManager in the initramfs to configure networking during early boot. Use the NetworkManager internal DHCP client for networking configuration instead. See BZ#1908462 for more information.

The current release stops updating the buildConfig.spec.triggers[i].imageChange.lastTriggeredImageID field when the ImageStreamTag referenced by buildConfig.spec.triggers[i].imageChage points to a new image. Instead, this release updates the buildConfig.status.imageChangeTriggers[i].lastTriggeredImageID field.

Additionally, the Build Image Change Trigger controller ignores the buildConfig.spec.triggers[i].imageChange.lastTriggeredImageID field.

Now, the Build Image Change Trigger controller starts a build based on the buildConfig.status.imageChangeTriggers[i].lastTriggeredImageID field and how it compares to the image ID now referenced by the ImageStreamTag referenced in the buildConfig.spec.triggers[i].imageChange.

Therefore, update scripts and jobs that inspect buildConfig.spec.triggers[i].imageChange.lastTriggeredImageID accordingly. (BUILD-190)

Support for using v1 without a group for apiVersion for OpenShift Container Platform resources has been removed. Every resource that includes *.openshift.io must match the apiVersion value found in the API index.

Starting with OpenShift Container Platform 4.9.24, support for using the Cloud Credential Operator (CCO) in mint mode on Microsoft Azure clusters has been removed from OpenShift Container Platform 4.9. This change is due to the planned retirement of the Azure AD Graph API by Microsoft on 30 June 2022 and is being backported to all supported versions of OpenShift Container Platform in z-stream updates.

For previously installed Azure clusters that use mint mode, the CCO attempts to update existing secrets. If a secret contains the credentials of previously minted app registration service principals, it is updated with the contents of the secret in kube-system/azure-credentials. This behavior is similar to passthrough mode.

For clusters with the credentials mode set to its default value of "", the updated CCO automatically changes from operating in mint mode to operating in passthrough mode. If your cluster has the credentials mode explicitly set to mint mode ("Mint"), you must change the value to "" or "Passthrough".

While the Azure AD Graph API is still available, the CCO in upgraded versions of OpenShift Container Platform attempts to clean up previously minted app registration service principals. Upgrading your cluster before the Azure AD Graph API is retired might avoid the need to clean up resources manually.

If the cluster is upgraded to a version of OpenShift Container Platform that no longer supports mint mode after the Azure AD Graph API is retired, the CCO sets an OrphanedCloudResource condition on the associated CredentialsRequest but does not treat the error as fatal. The condition includes a message similar to unable to clean up App Registration / Service Principal: <app_registration_name>. Cleanup after the Azure AD Graph API is retired requires manual intervention using the Azure CLI tool or the Azure web console to remove any remaining app registration service principals.

To clean up resources manually, you must find and delete the affected resources.