OpenShift Container Platform 4.9 release notes

GA: General Availability

TP: Technology Preview

DEP: Deprecated

REM: Removed

Previously, encryption conditions could remain indefinitely and be reported as a degraded condition for some Operators. Stale encryption conditions are now cleared properly and no longer improperly reported. (BZ#1974520)

Previously, the CA for API server client certificates was rotated early in the lifetime of a cluster, which prevented the Authentication Operator from creating a certificate signing request (CSR) because a previous CSR with the same name still existed. The Kubernetes API server was unable to authenticate itself to the OAuth API server when sending TokenReview requests, which caused authentication to fail. Generated names are now used when creating CSRs by the Authentication Operator, so an early rotation of the CA for API server client certificates no longer causes authentication failures. (BZ#1978193)

Previously, metal3 pods could not download an Red Hat Enterprise Linux CoreOS (RHCOS) image due to the sequencing of creating initContainers. This issue is fixed by reordering the creation of the initContainers, so that the metal-static-ip-set initContainer is created before the metal3-machine-os-downloader initContainer. The RHCOS image now downloads as expected. (BZ#1973724)

Previously, when using installer-provisioned installation on bare metal with a host configured to use idrac-virtualmedia, the bios_interface for that host got set to idrac-wsman by default. This resulted in the BIOS settings being unavailable and an exception occurring. This issue is fixed by using idrac-redfish for the default bios_interface when using idrac-virtualmedia. (BZ#1928816)

Previously, in UEFI mode, the ironic-python-agent created a UEFI bootloader entry after downloading the RHCOS image. When using an RHCOS image based on RHEL 8.4, the image could fail to boot using this entry and output a BIOS error screen. This is fixed by the ironic-python-agent configuring the boot entry based on a CSV file located in the image, instead of using a fixed boot entry. The image boots properly without error. (BZ#1966129)

Previously, if provisioningHostIP had been set in install-config, it was assigned to the metal3 pod, even in cases where the provisioning network had been disabled. This has been fixed. (BZ#1972753)

Previously, the assisted installer could not provision Supermicro X11/X12-based systems because of a mismatch in the sushy resource library. The mismatch resulted in an installation issue by being unable to attach virtual media to the Inserted and WriteProtected attributes, and not being allowed in the VirtualMedia.InsertMedia request body. This issue is fixed by modifying the sushy resource library, and adding a condition to stop sending these optional attributes when not strictly required, thus allowing the installation to progress past this point. (BZ#1986238)

Previously, some error types in the provisioned state caused the host to be deprovisioned. This occurred after a restart of the metal3 pod if the image provisioned to a bare metal host became unavailable. In this case, the host would enter the deprovisioning state. This issue is fixed by modifying the action of the error in the provisioned state so that if the image becomes unavailable, the error will be reported but deprovisioning will not be initiated. (BZ#1972374)

In OpenShift Container Platform and later, the fix for bug BZ#1884270 incorrectly pruned SSH protocol URLs in an attempt to provide SCP-styled URL capabilities. This error caused the oc new-build command not to pick an automatic source clone secret: the build could not use the build.openshift.io/sbuild.openshift.io/source-secret-match-uri-1ource-secret-match-uri-1 annotation to map SSH keys with the associated secrets, and therefore could not perform git cloning. This update reverts the changes from BZ#1884270 so that builds can use the annotation and perform git cloning.

Before this update, various allowed and block registry configuration options of the cluster image configuration could block the Cluster Samples Operator from creating image streams. When that happened, the samples operator marked itself as degraded, which impacted the general OpenShift Container Platform install and upgrade status.

Previously, when a root volume was created for a new server, and that server was not successfully created, the automatic deletion for the volume was not triggered because there was no deletion of a server associated with the volume. In some situations, this led to the creation of many extra volumes, and caused errors if the volume quota was reached. With this release, newly created root volumes are deleted when server creation call fails. (BZ#1943378)

Previously, when using the default value for instanceType, the Machine API created m4.large instances on AWS. This is different than the m5.large instance type for machines that are created by the OpenShift Container Platform installer. With this release, the Machine API creates m5.large instances for new machines on AWS when the default value is specified. (BZ#1953063)

Previously, the machine set definitions of compute nodes could not specify whether a port should be trunked. This was a problem for technologies that require the user to configure trunked and non-trunked ports for the same machines. This release adds a new field, spec.Port.Trunk = bool, which gives the user more flexibility to determine which ports result in trunks. If no value is specified, spec.Port.Trunk inherits the value of spec.Trunk and the name of the trunk created matches the name of the port used. (BZ#1964540)

Previously, the Machine API Operator constantly attached new targets even if they were already attached. The excessive calls to the AWS API resulted in a high number of errors. With this release, the Operator checks whether a load balancer attachment is required before attempting the attachment process. This change reduces the frequency of failed API requests. (BZ#1965080)

Previously, when using automatic pinning for a VM, the name of the property was disabled, existing, or adjust. With this release, the name better describes each policy, and existing was removed because it is blocked on oVirt. The new property names are none and resize_and_pin, which align with the oVirt user interface. (BZ#1972747)

Previously, the cluster autoscaler was unable to access the csidrivers.storage.k8s.io or csistoragecapacities.storage.k8s.io resources, which resulted in permissions errors. This fix updates the role assigned to the cluster autoscaler so that it includes permissions for these resources. (BZ#1973567)

Previously, it was possible to delete a machine with a node that has been deleted. This caused the machine to remain in a deleting phase indefinitely. This fix allows you to delete machines in this state properly. (BZ#1977369)

When using boot-from-volume image, creating a new instance leaks volumes if the machine controller is rebooted. This caused the previously created volume to never be cleaned up. This fix ensures that the volume created previously is either pruned or reused. (BZ#1983612)

Previously, the Red Hat Virtualization (RHV) provider ignored NICs with br-ex names for machines. Since a network type of OVNKubernetes creates a NIC with a br-ex name, this resulted in the machine never getting an IP address on OVN-Kubernetes. With this fix, it is now possible to install OpenShift Container Platform on RHV with network set to OVNKubernetes. (BZ#1984481)

Previously, when deployed on Red Hat OpenStack Platform (RHOSP) with a combination of proxy and custom CA certificate, a cluster would not become fully operational. This fix passes the proxy settings to the HTTP transport used when connecting with a custom CA certificate, ensuring that all cluster components work as expected. (BZ#1986540)

Previously, the Cluster Version Operator (CVO) did not respect the noProxy property in the proxy configuration resource. As a result, the CVO was denied access to update recommendations or release signatures when only unproxied connections completed. Now, when the proxy resource requests direct, unproxied access, the CVO reaches the upstream update service and signature stores directly. (BZ#1978749)

Previously, the Cluster Version Operator (CVO) loaded its proxy configuration from proxy resource specification properties instead of from status properties that were verified by the Network Operator. As a result, any incorrectly configured values would prevent the CVO from reaching the upstream update service or signature stores. Now, the CVO loads its proxy configuration only from the verified status properties. (BZ#1978774)

Previously, the Cluster Version Operator (CVO) did not remove volume mounts that were added outside of the manifest. As a result, pod creation could fail during a volume failure. Now, CVO removes all volume mounts that do not appear in the manifest. (BZ#2004568)

Previously, when working with Ceph storage, the Console Storage Plugin unnecessarily included a redundant use of a namespace parameter. This bug had no customer-visible impact; however, the plugin has been updated to avoid the redundant use of the namespace. (BZ#1982682)

The Operator to check if the registry should use custom tolerations was checking spec.nodeSelector instead of spec.tolerations. The custom tolerations from spec.tolerations are applied only when spec.nodeSelector is set. This fix uses the field spec.tolerations to check for the presence of custom tolerations. Now, the Operator uses custom tolerations if spec.tolerations are set. (BZ#1973318)

The spec.managementState in configs.imageregistry is set to Removed, which caused the image pruner pod to generate warnings about deprecated CronJob in v1.21 and later, and that batch/v1 should be used. This fix updates batch/v1beta1 with batch/v1 in OpenShift Container Platform oc. Now, warnings about the deprecated CronJob in image pruner pods no longer appear. (BZ#1976112)

Previously, the network interface on Azure control plane nodes was missing a hyphen in the interface name. This was inconsistent compared to other platforms, which caused issues. The missing hyphen has been added. Now all control plane nodes are named the same, regardless of the platform. (BZ#1882490)

You can now configure the autoPinningPolicy and hugepages fields in the install-config.yaml file for oVirt. The autoPinningPolicy field allows you to automatically set the non-uniform memory access (NUMA) pinning settings and CPU topology changes for the cluster. The hugepages field allows you to set the Hugepages of the hypervisor. (BZ#1925203)

Previously, the installation program did not output any errors when the Ed25519 SSH key type was used with FIPS enabled, even though it could not be used. Now the installation program validates SSH key types, outputting an error when an SSH key type is not supported with FIPS enabled; only RSA and ECDSA SSH key types are allowed when FIPS is enabled. (BZ#1962414)

In certain conditions, Red Hat OpenStack Platform (RHOSP) network trunks do not contain a tag to indicate that the trunk belongs to the cluster. Consequently, cluster deletion missed the trunk ports and got stuck in a loop until they timed out. Deleting the cluster now deletes trunks for which the tagged port is a parent. (BZ#1971518)

Previously, when uninstalling a cluster on Red Hat OpenStack Platform (RHOSP), the installer used an inefficient algorithm to delete resources. The inefficient algorithm caused the uninstall process to require more time than was necessary. The installer is updated with a more efficient algorithm that should uninstall the cluster more quickly. (BZ#1974598)

Previously, if the AWS_SHARED_CREDENTIALS_FILE environment variable was set to an empty file, the installer prompted for credentials and then created a aws/credentials file, ignoring the value of the environment variable and possibly overwriting existing credentials. With this fix, the installer is updated to store credentials in the specified file. If the specified file has invalid credentials, the installer produces an error instead of overwriting the file and risking information loss. (BZ#1974640)

Previously, users encountered a vague error message when they deleted a cluster on Azure that shared resources with another cluster, making it difficult to understand why the deletion failed. This update adds an error message that explains why the failure occurs. (BZ#1976016)

Previously, because of a typo, Kuryr deployments were being checked against the wrong requirements, meaning that installations with Kuryr could succeed even if they did not meet the minimum requirements for Kuryr. This fix eliminates the error, allowing the installer to check the right requirements. (BZ#1978213)

Before this update, the ingress checks for keepalived did not include fall and raise directives, which meant that a single failed check could cause an ingress virtual IP failover. This bug fix introduces fall and raise directives to prevent such failovers. (BZ#1982766)

Previously, when a deployment and image stream were created at the same time, a race condition could occur which caused the deployment controller to create replica sets in an infinite loop. The responsibilities of the API server’s image policy plugin were lowered and concurrent creation of a deployment and image stream no longer causes infinite replica sets. (BZ#1925180), (BZ#1976775)

Previously, there was a race between the installer pod and the cert-syncer container, which were writing to the same path. This could leave some certificates empty and prevent the server from running. Kubernetes API server certificates are now written in an atomic way to prevent races between multiple processes. (BZ#1971624)

Before this update, even though the aim of the cluster-monitoring-view user role was to only allow access to Alertmanager, non-administrator users assigned to this role could still create and silence alerts. With this update, users assigned only to this role can no longer create or silence alerts. To allow non-administrator users to create and silence alerts, you must assign them the new monitoring-alertmanager-edit role in addition to the cluster-monitoring-view role. (BZ#1947005)

When using the OVN-Kubernetes cluster network provider, the logical flow cache was configured without any memory limit. As a result, in some situations high memory pressure could cause a node to become unusable. With this update, the logical flow cache is configured with a 1 GB memory limit by default. (BZ#1961757)

When using the OVN-Kubernetes cluster network provider, any network policies created in a OpenShift Container Platform 4.5 cluster that was subsequently upgraded might allow or drop unexpected traffic. In later versions of OpenShift Container Platform, OVN-Kubernetes uses a different convention for managing IP address sets, and any network policies created in OpenShift Container Platform 4.5 did not use this convention. Now, during upgrades all network policies are migrated to the new convention. (BZ#1962387)

For the OVN-Kubernetes cluster network provider, when using must-gather to retrieve Open vSwitch (OVS) logs, the INFO log level was missing from the gathered logging data. Now all log levels are included in OVS logging data. (BZ#1970129)

Previously, performance testing revealed that the service controller metrics had a significant increase in cardinality due to a label requirement. As a result, memory usage was elevated on Open Virtual Network (OVN) Prometheus pods. With this update, the label requirement is removed. Service controller cardinality metrics and memory usage are now reduced. (BZ#1974967)

Previously, ovnkube-trace required iproute to be installed in the source and/or destination pod because it needed to detect the interfaces link index. This causes ovnkube-trace to fail on pods if there is no iproute installed. Now, you can get the link index from /sys/class/net/<interface>/iflink instead of iproute. As a result, ovnkube-trace no longer requires iproute to be installed in source and destination pods. (BZ#1978137)

Previously, the Cluster Network Operator (CNO) deployed a service monitor for the network-check-source service to get discovered by Prometheus without correct annotations and role-based access control (RBAC). As a result, the service and its metrics never populated in Prometheus. Now, the correct annotations and RBAC are added to the namespace of network-check-source service. Now, metrics of service network-check-source get scraped by Prometheus. (BZ#1986061)

Previously, when using IPv6 DHCP, node interface addresses might be leased with a /128 prefix. Consequently, OVN-Kubernetes uses the same prefix to infer the node’s network and routes any other address traffic, including traffic to other cluster nodes, through the gateway. With this update, OVN-Kubernetes inspects the node’s routing table and checks for the wider routing entry for the node’s interface address and uses that prefix to infer the node’s network. As a result, traffic to other cluster nodes is no longer routed through the gateway. (BZ#1980135)

Previously, when a cluster used the OVN-Kubernetes Container Network Interface provider, attempting to add an egress router with IPv6 address failed. With the fix, support for IPv6 is added to the egress router CNI plugin and adding adding egress routers succeeds. (BZ#1989688)

Previously, in containers, CRI-O did not create a symbolic link from /proc/mounts file to the /etc/mtab file. As a consequence, the user could not view the list of the mounted devices in the container’s /etc/mtab file. CRI-O now adds the symbolic link. As a result, users can view the container’s mounted devices. (BZ#1868221)

Previously, if pods were deleted quickly after creation, the kubelet might not clean up the pods properly. This resulted in pods being stuck in a terminating state, and could impact the availability of upgrades. This fix improves the pod lifecycle logic to avoid this problem. (BZ#1952224)

Previously, the SystemMemoryExceedsReserved alert would fire when the system memory usage exceeded 90% of the reserved memory. As a result, clusters could fire an excessive number of alerts. The threshold for this alarm was changed to fire at 95% of reserved memory. (BZ#1980844)

Previously, a bug in CRI-O caused CRI-O to leak a child PID of a process it created. As a result, if under load, systemd could create a significant number of zombie processes. This could lead to node failure if the node ran out of PIDs. CRI-O was fixed to prevent the leakage. As a result, these zombie processes are no longer being created. (BZ#2003197)

Previously, the oc command-line tool was crashing while mirroring the registry, causing a slice bounds out of range panic runtime error because of an unchecked index operation on a slice when using the --max-components argument. With this fix, a check has been added to ensure that the components check does not request an out-of-range index value so that the oc tool no longer panics when using the --max-components argument. (BZ#1786835)

Previously, the oc describe quota command showed inconsistent units in Used memory for the ClusterResourceQuota value, which was unpredictable and difficult to read. With this fix, the Used memory now always uses the same unit as the Hard memory so that the oc describe quota command shows predictable values. (BZ#1955292)

Previously, the oc logs command did not work with pipeline builds because of a missing client setup. The client setup has been fixed in the oc logs command so that it now works with pipeline builds. (BZ#1973643)

Previously, the Operator Lifecycle Manager (OLM) upgradeable condition message was unclear when installed Operators set olm.maxOpenShiftVersion to a minor OpenShift Container Platform version less than or equal to the current version. This resulted in an incorrect error message that has been fixed to specify that only minor and major version upgrades are blocked when olm.maxOpenShiftVersion is set to version different than the current OpenShift Container Platform version. (BZ#1992677)

Previously, the opm command failed to deprecate bundles when they were present in the index. Consequently, bundles truncated as part of another deprecation in the same call were reported as missing. This update adds a check for bundles before any deprecation takes place to differentiate between a bundle that is not present and one that has been truncated. As a result, deprecated bundles along the same upgrade path are no longer reported as missing. (BZ#1950534)

A transient error can occur when Operator Lifecycle Manager (OLM) attempts to update a custom resource definition (CRD) object in the cluster. This caused OLM to permanently fail the install plan containing the CRD. This bug fix updates OLM to retry CRD updates on resource-modified conflict errors. As a result, OLM is now more resilient to this class of transient errors. Install plans no longer permanently fail on conflict errors that OLM is able to retry and resolve. (BZ#1923111)

The opm index|registry add commands attempted to verify the existence of Operator bundles in an index that are replaced, regardless of whether they were already truncated from the index. The commands would consistently fail after a bundle was deprecated for a given package. This bug fix updates the opm CLI to handle this edge case and no longer verify the existence of truncated bundles. As a result, the commands no longer fail after a bundle is deprecated for a given package. (BZ#1952101)

Operator Lifecycle Manager (OLM) can now allow priority classes to be projected into registry pods using labels in catalog source resources. The default catalog sources are important components in namespaces managed by the cluster, which mandate priority classes. With this enhancement, all default catalog sources in the openshift-marketplace namespace have a system-cluster-critical priority class. (BZ#1954869)

The Marketplace Operator was using the leader-for-life implementation where a config map holding the leasing owner’s identity has owner references placed by the controller’s pod. This is problematic in the case where the node that the pod was scheduled on became unavailable, and the pod was unable to be terminated. This made the config map unable to be properly garbage collected so a new leader could be elected. Minor version cluster upgrades were blocked as the newer Marketplace Operator version could not gain leader election. Manual cleanup of the config map holding the leader election lease was required in order to release the lock and complete the upgrade of the Marketplace component. This bug fix switches to using the leader-for-lease leader election implementation. As a result, leader election no longer gets stuck in this scenario. (BZ#1958888)

Previously, a new Failed phase was introduced for install plans. Failure to detect a valid Operator group (OG) or service account (SA) resource for the namespace the install plan was being created in would transition the install plan to the failed state. That is, failure to detect these resources when the install plan was reconciled the first time was considered a permanent failure. This was a regression from the following previous behavior of install plans:

conditions:
 - lastTransitionTime: ""2021-06-23T18:16:00Z""
 lastUpdateTime: ""2021-06-23T18:16:16Z""
 message: attenuated service account query failed - no operator group found that
 is managing this namespace
 reason: InstallCheckFailed
 status: ""False""
 type: Installed

conditions:
 - lastTransitionTime: ""2021-06-23T18:33:37Z""
 lastUpdateTime: ""2021-06-23T18:33:37Z""
 status: ""True""

When updating a catalog source, a Get call is immediately followed by a Delete call on a number of resources related to the catalog source. In some instances, the resource had already been deleted but the resource still existed in the cache. This allowed the Get call to succeed, but the following Delete call failed as the resource did not exist on the cluster. This bug fix updates Operator Lifecycle Manager (OLM) to ignore the error returned by the Delete call if the resource is not found. As a result, OLM no longer reports an error when updating a catalog source due to a caching issue that results in a "Resource Not Found" error from the Delete call. (BZ#1967621)

A cluster service version (CSV) with a name over the limit of 63 characters causes an invalid ownerref label. Previously, when Operator Lifecycle Manager (OLM) used the ownerref reference to retrieve owned resources, including cluster role bindings, the lister returned all cluster role bindings in the namespaces due to the invalid label. This bug fix updates OLM to use a different method to let the server reject invalid ownerref labels instead. As a result, when CSVs have an invalid name, OLM no longer removes cluster role bindings. (BZ#1970910)

Previously, Operator dependencies were not always persisted after installation time. After installing an Operator that declares dependencies, later updates and installations within the same namespace could fail to honor the previously installed Operator’s dependencies. With this bug fix, dependencies are now persisted, along with all declared properties for the Operator, in an annotation on the Operator’s ClusterServiceVersion (CSV) object. As a result, the declared dependencies of installed Operators continue to be respected for future installations. (BZ#1978310)

Previously, when you removed an Operator with a deprecated bundle, the deprecation history was not included in the garbage collection. As a result, if you reinstalled the Operator, the bundle version would show up the deprecated table. This update fixes the issue with better garbage collection for deprecated bundles. (BZ#1982781)

Previously, the z-stream version of a cluster was used in Operator compatibility calculations. As a result, micro releases of OpenShift Container Platform were blocked. This update fixes the issue by ignoring cluster z-stream versions in Operator compatibility comparisons. (BZ#1993286)

Previously, a single failed request to the discovery endpoint of a service could make an Operator report Available=False. To increase resilience, a set of improvements have been introduced to prevent some Operators from reporting Available=False during an update due to various transient errors. (BZ#1948089)

Previously, when creating an update service application through the web console, an invalid host error occurred. This occurred because the default OpenShift Update Service (OSUS) application name was too long. A shorter default name is now in place and the error no longer occurs. (BZ#1939788)

Previously, the Performance Addon Operator could not restart properly in environments with a bandwidth-limited connection. It also could not restart properly when single node clusters, or any other edge nodes, lost connection to the image registry. With this update, the issue is resolved by ensuring the image is not pulled from registry.redhat.io if the image is already available on the node. This fix ensures that the Performance Addon Operator restarts correctly using the image from the local image cache. (BZ#2055019)

Previously, systemd was unable to read environment files in /etc/kubernetes. The SELinux policy caused this and as a result, the kubelet did not start. The policy has been modified. The kubelet starts and the environment files are read. (BZ#1969998)

In an s390x Kernel Virtual Machine (KVM) with an ECKD DASD attached, the DASD would appear to be a regular virtio storage device but would become inaccessible if the VTOC was removed. As a result, you could not use DASD as a virtio block device when installing Red Hat Enterprise Linux CoreOS (RHCOS) on the KVM. The coreos-installer program has been updated so that it now installs Red Hat Enterprise Linux CoreOS (RHCOS) with a VTOC-format partition table when the installation destination is a virtio storage device such as an ECKD DASD attached to a KVM. (BZ#1960485)

Previously, the NetworkManager-wait-online-service timed out too early, which prevented establishing a connection before the coreos-installer program could start. Consequently, if the network took too long to start, the coreos-installer program failed to fetch the Ignition config. With this update, the NetworkManager-wait-online-service timeout has been increased to its default upstream value. As a result, the coreos-installer program no longer fails to fetch the Ignition config. (BZ#1967483)

Previously, there was config drift when the Cluster Network Operator (CNO) attempted to sanitize the proxy configuration, specifically the no_proxy config. This resulted in a specific IPv6 CIDR missing from the no_proxy. This fix implements logic that updates the dual stack (IPV4 and IPV6) for all scenarios. (BZ#1981975)

Previously, if the .spec.privateZone field of the dns.config.openshift.io Operator was filled out incorrectly so that the Ingress Operator was not able to find the private hosted zone, then the Ingress Operator became degraded. However, even after fixing the .spec.privateZone field, the Ingress Operator stayed degraded. The Ingress Operator finds the hosted zone and adds the .apps resource record, but the Ingress Operator does not reset the degraded status. This fix watches the DNS config object and monitors changes regarding the spec.privateZone field. It applies the appropriate logic and updates the Operator status accordingly. The Operator status returns to degraded, or False, once the correct .spec.privateZone field is set. (BZ#1942657)

Previously, the lack of a connection timeout led to lengthy delays. This occurred when the Cluster Samples Operator, with managementState set to Removed, tested the connection to registry.redhat.io. With the addition of a connection timeout, the delay is eliminated. (BZ#1990140)

Previously, you could delete LocalVolumeSets with in-use PVs, which required manual clean up. This fix ensures that all released PVs are cleaned automatically. (BZ#1862429)

Previously, the oc get volumesnapshotcontent command did not display the namespace for a volume snapshot, which meant that the volume snapshot was not uniquely identified. This command now displays the namespace for the volume snapshot. (BZ#1965263)

Previously, the Manila CSI Operator used a custom transport when communicating with a Red Hat OpenStack Platform (RHOSP) endpoint that used self-signed certificates. Because this custom transport did not consume the proxy environment variables, the Manila CSI Operator would fail to communicate with Manila. This update ensures that the custom transport consumes the proxy environment variables. As a result, the Manila CSI Operator now works with proxy and custom CA certificates. (BZ#1960152)

Previously, the Cinder CSI Driver Operator was not using the configured proxy to connect to the Red Hat OpenStack Platform (RHOSP) API, which could cause the installation to fail. With this update, an annotation is included in the Cinder CSI Driver Operator deployment that ensures proxy environment variables are set on the container. As a result, the installation no longer fails. (BZ#1985391)

The frequency at which the Local Storage Operator inspects newly added block devices has been changed from 5 seconds to 60 seconds to decrease its CPU consumption. (BZ#1994035)

Previously, communication failure with the Manila CSI Operator would degrade the cluster. With this update, failed communication with the Manila CSI Operator endpoint results in a non-fatal error. As a result, the Manila CSI Operator gets disabled instead of degrading the cluster. (BZ#2001958)

Previously, the Local Storage Operator deleted orphaned persistent volumes (PVs) with a 10 second delay, and the delay was cumulative. When several persistent volume claims (PVCs) were deleted at the same time, it could take several minutes or hours to get their PVs deleted. Consequently, corresponding local disks were not available for new PVCs for several hours. With this fix, the 10 second delay is removed. As a result, PVs are detected and corresponding local disks are made available for new PVCs sooner. (BZ#2007684)

Previously, all rows on the PF4 table were rerendering. With this update, the content in React.memo was wrapped so the content does not rerender on every scroll event. (BZ#1856355)

Previously, the charts in Cluster Utilization in the OpenShift Container Platform web console displayed the data time span in a confusing manner. For example, if the six-hour time span option is selected, but data exists only for the final three hours, those three data points are stretched to fill the entire chart. The first three hours are not displayed. This could result in the assumption that the chart is showing the full six-hour time span. To avoid confusion, the charts now show blank space for missing information. In this example, the chart displays the entire six-hour time span with data starting at the fourth hour. The first three hours are blank. (BZ#1904155)

Previously, NetworkPolicy was not translated to Korean or Chinese on the web console. With this update, NetworkPolicy is now translated correctly when viewing the web console in Korean or Chinese. (BZ#1965930)

Previously, an issue with the Needs Attention state of the Console Overview section showed Operators as upgrading, even if they were not upgrading. This update fixes the Needs Attention state so that the correct status of an Operator is displayed. (BZ#1967047)

Previously, the alert for a failed Cluster Service Version (CSV) displayed a generic status.message that did not help troubleshoot the failed CSV. With this update, copied CSVs show a helpful message and a link to the original CSV to troubleshoot. (BZ#1967658)

Previously, a user was unable to use the drop-down options in the masthead with a keyboard. With this update, users are now able to access the drop-down options with a keyboard. (BZ#1967979)

Previously, a utility function used for matching Operator-owned resources with their owners would return false matches. Consequently, Managed by links on Operator-owned resource pages would sometimes link to the incorrect URL. This fix updates the function logic to correctly match owned Operators. As a result, Managed by links now link to the correct URL. (BZ#1970011)

Previously, the OperatorHub web console interface would lead users to unrelated install plans. With this update, OperatorHub links users to the Operator Subscription details tab to view installation progress. (BZ#1970466)

Previously, items in the Add drop-down list on the OAuth details page were not internationalized. With this update, these items are internationalized and the user experience for non-English speakers is improved. (BZ#1970604)

Previously, an invalid localization property prevented some messages from being internationalized. This update removes the invalid property. As a result, these messages are internationalized and the user experience for non-English speakers is improved. (BZ#1970980)

This update removes a tooltip that appeared when mousing over a resource link on a list page, because the information did not improve the user experience. (BZ#1971532)

Previously, console pods were deployed with the preferredDuringSchedulingIgnoredDuringExecution anti-affinity rule, which sometimes resulted in both console pods being scheduled on the same control plane node. This fix changes the rule to requiredDuringSchedulingIgnoredDuringExecution so that the pods must be scheduled on different nodes if the condition matches. (BZ#1975379)

Previously, uninstalling an Operator failed to remove all of the enabled plugins. With this release, uninstalling an Operator now removes all enabled plugins. (BZ#1975820)

Previously, front-end Operator Lifecycle Manager (OLM) descriptor handling only used the first x-descriptor to render a property on an operand details page. Consequently, if multiple x-descriptors were defined for a property and the first one in the list was invalid or unsupported, it would not render as expected. This fix updates the descriptor validation logic to prioritize supported x-descriptors over unsupported x-descriptors. As a result, descriptor-decorated properties are rendered on the Operand details page using the first valid and supported x-descriptor found in the list. (BZ#1976072)

Previously, string data was used for encoded secrets. As a result, binary secret data was not properly uploaded by the web console. This update encodes secrets and uses data instead of string data in the API. As a result, binary secrets now upload correctly. (BZ#1978724)

Previously, when processes running on the cluster were manually terminated, the terminal ps -aux command showed that some processes were not cleared. This caused stray processes to remain, leaving the cluster in an invalid state. This fix ensures that all processes terminate properly on the cluster and do not appear in the list of active processes that are listed on the terminal. (BZ#1979571)

Previously, when a default pull secret was added to a new project and credentials for multiple registries were uploaded, only the first credential was listed on the Project Details page. There was also no indication that the list was truncated. As a result, when a user clicked the project details from Default pull secret, only the first credential was listed. This fix ensures that all of the credentials are listed and informs the users that additional credentials exist if they are not listed on the current page. (BZ#1980704)

Previously, when users changed the default browser language to Simplified Chinese, the cluster utilization resource metrics on the Overview page of the web console displayed in a combination of English and Simplified Chinese characters. This fix allows the user to view the cluster utilization resources entirely in the selected language. (BZ#1982079)

Previously, when the language was changed to Simplified Chinese, the cluster utilization usage statistic did not match the translation in the left menu for project, pod, and node. This fix corrects the Simplified Chinese translation so that the cluster utilization metrics are consistent with the top consumers filter. (BZ#1982090)

Previously, users saw an error instead of the default pull secret from the service account. This resulted in incomplete information on the project details screen. The user had to go to the default ServiceAccount to view the entire list of default pull secrets. This fix allows the user to view the entire list of pull secrets from the default ServiceAccount on the project details page. (BZ#1983091)

Previously, if you resized the web page for a node or pod while viewing the Terminal tab, sometimes the browser displayed two vertical scrollbars. The console is now updated to display one scrollbar only when the window is resized. (BZ#1983220)

Previously, the web console did not deploy when installing OpenShift Container Platform 4.8.2 using a single node developer profile. If a valid Operator group or service account was not detected for the namespace in which the install plan was being created, the install plan was placed in a failed state. No further attempts were made. With this revision, the failed install plan is set to run again until an Operator group or service account is detected. (BZ#1986129)

Previously, in the Events Dashboard, More and Show Less were not internationalized, which resulted in poor user experience. With this update, the text is now internationalized. (BZ#1986754)

Previously, the logic that constructed the fully qualified domain name (FQDN) of a service in the Console page was missing. As a result, FQDN information was missing on the service’s detail page. This update adds logic that constructs the FQDN so that the service’s FQDN information is now available on the page. (BZ#1996816)

Previously, kamelets of type sink were shown in the catalog for event sources along with source kamelets. In the current release, the catalog for event sources displays only kamelets of type source. (BZ#1971544)

Previously, the log file contained information in a single line without any line breaks. In the current release, the log file contains the expected line breaks with additional line breaks around log headers. (BZ#1985080)

TP: Technology Preview

GA: General Availability

-: Not Available

DEP: Deprecated

In OpenShift Container Platform 4.1, anonymous users could access discovery endpoints. Later releases revoked this access to reduce the possible attack surface for security exploits because some discovery endpoints are forwarded to aggregated API servers. However, unauthenticated access is preserved in upgraded clusters so that existing use cases are not broken.

## Snippet to remove unauthenticated group from all the cluster role bindings
$ for clusterrolebinding in cluster-status-binding discovery system:basic-user system:discovery system:openshift:discovery ;
do
### Find the index of unauthenticated group in list of subjects
index=$(oc get clusterrolebinding ${clusterrolebinding} -o json | jq 'select(.subjects!=null) | .subjects | map(.name=="system:unauthenticated") | index(true)');
### Remove the element at index from subjects array
oc patch clusterrolebinding ${clusterrolebinding} --type=json --patch "[{'op': 'remove','path': '/subjects/$index'}]";
done

When upgrading to OpenShift Container Platform 4.9, the Cluster Version Operator blocks the upgrade for approximately five minutes while failing precondition checks. The error text, which says It may not be safe to apply this update, might be misleading. This error occurs when one or multiple precondition checks fail. In some situations, these precondition checks might only fail for a short period of time, for example, during an etcd backup. In these situations, the Cluster Version Operator and corresponding Operators will, by design, automatically resolve the failing precondition checks and the CVO successfully starts the upgrade.

The oc annotate command does not work for LDAP group names that contain an equal sign (=), because the command uses the equal sign as a delimiter between the annotation name and value. As a workaround, use oc patch or oc edit to add the annotation. (BZ#1917280)

Cluster administrators can specify a custom HTTP error code response page for either 503, 404, or both error pages. If you do not specify the correct format for the custom error code response page, a router pod outage occurs and does not resolve. The router does not reload to reflect custom error code page updates. As a workaround, you can use the oc rsh command to locally access the router pods. Then run reload-haproxy in all the router pods that serve the custom http error code pages:

$ oc -n openshift-ingress rsh router-default-6647d984d8-q7b58
sh-4.4$ bash -x /var/lib/haproxy/reload-haproxy

An Open Virtual Network (OVN) bug causes persistent connectivity issues with Octavia load balancers. When Octavia load balancers are created, OVN might not plug them into some Neutron subnets. These load balancers might be unreachable for some of the Neutron subnets. This problem affects Neutron subnets, which are created for each OpenShift namespace at random when Kuryr is configured. As a result, when this problem occurs the load balancer that implements OpenShift Service objects will be unreachable from OpenShift namespaces affected by the issue. Because of this bug, OpenShift Container Platform 4.8 deployments that use Kuryr SDN are not recommended on Red Hat OpenStack Platform (RHOSP) 16.1 with OVN and OVN Octavia configured. This will be fixed in a future release of RHOSP. (BZ#1937392)

Installations on Red Hat OpenStack Platform (RHOSP) with Kuryr will not work if configured with a cluster-wide proxy when the cluster-wide proxy is required to access RHOSP APIs. (BZ#1985486)

Due to a race condition, the Red Hat OpenStack Platform (RHOSP) cloud provider might not start properly. Consequently, LoadBalancer services might never get an EXTERNAL-IP set. As a temporary workaround, you can restart the kube-controller-manager pods using the procedure described in (BZ#2004542).

The ap-northeast-3 AWS region is not provided as an option by the installation program when installing OpenShift Container Platform, even though it is a supported AWS region. As a temporary workaround, you can select a different AWS region from the installation prompt and then update the region information in the generated install-config.yaml file before installing your cluster. (BZ#1996544)

When installing a cluster on AWS in the us-east-1 region, you cannot use local AWS zones. As a temporary workaround, you must specify non-local availability zones in the install-config.yaml file when installing a cluster. (BZ#1997059)

You can only install OpenShift Container Platform on Azure Stack Hub with public endpoints, such as the ARM endpoint, that are secured with certificates signed by a publicly trusted certificate authority (CA). Support for internal CAs will be added in a future z-stream release of OpenShift Container Platform. (BZ#2012173)

Cluster administrators can specify a custom HTTP error code response page for either 503, 404, or both error pages. The router does not reload to reflect custom error code pages updates. As a workaround, rsh in the router pods and run reload-haproxy in all the router pods that serve the custom http error code pages:

$ oc -n openshift-ingress rsh router-default-6647d984d8-q7b58
sh-4.4$ bash -x /var/lib/haproxy/reload-haproxy

This release contains a known issue. If you customize the hostname and certificate of the OpenShift OAuth route, Jenkins no longer trusts the OAuth server endpoint. As a result, users cannot log in to the Jenkins console if they rely on the OpenShift OAuth integration to manage identity and access. A workaround is not available at this time. (BZ#1991448)

Because certain high cardinality monitoring metrics were inadvertently dropped, the following container performance input and output metrics are not available in this release: pod, qos, and System.

The Special Resource Operator (SRO) might fail to install on Google Cloud Platform due to a software-defined network policy. As a result, the simple-kmod pod is not created. This is fixed in OpenShift Container Platform 4.9.4 release. (BZ#1996916)

In OpenShift Container Platform 4.9, a user with cluster role cannot scale a deployment or deployment config using the console if they do not have edit rights to the deployment or deployment config. (BZ#1886888)

In OpenShift Container Platform 4.9, when minimal or no data exists in the Developer Console, most of the monitoring charts or graphs (for example, CPU consumption, memory usage, and bandwidth) show a range of -1 to 1. However, none of these values can ever go below zero, so the negative values are incorrect. (BZ#1904106)

The ip vrf exec command does not work due to a cgroups mismatch. As a result, this command cannot be used inside OpenShift pods. To use virtual routing and forwarding (VRF), applications must be VRF-aware and bind directly to the VRF interface. (BZ#1995631)

A nonuniform memory access (NUMA) bug can cause undesired NUMA pinning for the container, which can lead to latency or performance degradation. The Topology Manager can pin the container, with resources that the single-numa-node topology management policy can satisfy, to more than one NUMA node. The container is pinned under the guaranteed Quality of Service (QoS) pod. As a workaround, do not start guaranteed QoS pods when the container memory-resource requests are bigger than the single-numa-node policy can provide. (BZ#1999603)

Occasionally, a pod selected for deletion is not deleted. This occurs when a cluster is running out of resources. To reclaim resources, the system selects one or more pods for deletion. With low resources causing slow processing, the deletion operation may exceed the established grace period for deletion, resulting in failure. If this occurs, manually delete the pod. The cluster then reclaims the freed up resources. (BZ#1997476)

Intermittently, pods can hang in the ContainerCreating state and time out while waiting for Open vSwitch (OVS) port binding. The reported event is failed to configure pod interface: timed out waiting for OVS port binding. This issue can occur when many pods are created for the OVN-Kubernetes plugin. (BZ#2005598)

After rebooting the egress node, the lr-policy-list contains errors, such as duplicate records or missed internal IP addresses. The expected result is that the lr-policy-list has the same records as before rebooting the egress node. As a workaround, you can restart the ovn-kubemaster pods. (BZ#1995887)

When IP multicast relay is enabled on a logical router that contains distributed gateway ports, multicast traffic is not forwarded correctly on the distributed gateway port. The result is broken IP multicast functionality in OVN-Kubernetes. (BZ#2010374)

In the Administrator perspective of the web console, a page that is supposed to display a list of nodes is rendered before the list of nodes is available, which causes the page to become unresponsive. There is no workaround, but this issue will be addressed in a future release. (BZ#2013088)

Operator Lifecycle Manager (OLM) uses a combination of timestamp checks and obsolete API calls, which do not work for skipRange upgrades, to determine if it is necessary to perform an upgrade for a particular subscription. For Operators that use the skipRange upgrade, there is a delay in the upgrade process that can take up to 15 minutes to resolve and can potentially be blocked for even longer.

Currently, when launching Red Hat Enterprise Linux (RHEL) 8 on Google Cloud Platform with FIPS mode enabled, RHEL 8 fails to download metadata when trying to install packages from the Red Hat Update Infrastructure (RHUI). As a temporary workaround, you can disable RHUI repositories and use Red Hat Subscription Management to get content. (BZ#2001464), (BZ#1997516).

Following an OpenShift Container Platform single node reboot, all pods are restarted which causes significant load and longer than normal pod creation times. This happens because the Container Network Interface (CNI) is not able to process the pod add events quickly enough. The following error message is displayed: timed out waiting for OVS port binding. The OpenShift Container Platform single node instance eventually recovers, just slower than expected. (BZ#1986216)

When MetalLB is run in layer 2 mode with the OVN-Kubernetes Container Network Interface network provider, instead of a single node with a speaker pod responding to an ARP or NDP request, every node in the cluster responds to the request. The unexpected number of ARP responses might resemble an ARP-spoofing attack. Although the experience is different than designed, traffic is routed to the service as long as no software on the hosts or subnet is configured to block ARP. This bug is fixed in a future OpenShift Container Platform release. (BZ#1987445)

When Tang disk encryption and a static IP address configuration are applied on a VMWare vSphere user-provisioned infrastructure cluster, the nodes fail to boot properly after they are initially provisioned. (BZ#1975701)

Operators must list any related images for Operator Lifecycle Manager (OLM) to run from a local source. Presently, if the relatedImages parameter of the ClusterServiceVersion (CSV) object is not defined, opm render does not populate the related images. This is planned to be fixed in a later release. (BZ#2000379)

Previously, Open vSwitch (OVS) ran in a container on each OpenShift Container Platform cluster node and the node exporter agent collected OVS CPU and memory metrics from the nodes. Now, OVS runs on the cluster nodes as systemd units and the metrics are not collected. This is planned to be fixed in a later release. OVS packet metrics are still collected. (BZ#2002868)

The flag that is used to hide or show the Storage → Overview page in the OpenShift Container Platform web console is misconfigured. As a result, the overview page is not visible after deploying a cluster that includes OpenShift Cluster Storage. This is planned to be fixed in a later release. (BZ#2013132)

In OpenShift Container Platform 4.6 and later, image references for a pull must specify the following Red Hat registries:

Prior to OpenShift Container Platform 4.8 the default load balancing algorithm was leastconn. The default was changed to random in OpenShift Container Platform 4.8.0 for non-passthrough routes. Switching to random is incompatible with environments that need to use long-running websocket connections because it significantly increases memory consumption in those environments. To mitigate this significant memory consumption, the default load balancing algorithm was reverted to leastconn in OpenShift Container Platform 4.9. Once there is a solution that does not incur significant memory usage, the default will be changed to random in a future OpenShift Container Platform release.

$ oc get deployment -n openshift-ingress router-default -o yaml | grep -A 2 ROUTER_LOAD_BALANCE_ALGORITHM
        - name: ROUTER_LOAD_BALANCE_ALGORITHM
          value: leastconn

$ oc annotate -n <NAMESPACE> route/<ROUTE-NAME> "haproxy.router.openshift.io/balance=random"

The oc adm release extract --tools command fails when an image that is hosted in the local registry is specified. (BZ#1823143)

On an OpenShift Container Platform single node configuration, pod creation times are over two times slower when using the real-time kernel (kernel-rt) than when using the non-real time kernel. When using kernel-rt, the slower pod creation times affect the maximum number of supported pods because recovery time is impacted after a node reboots.

Following an OpenShift Container Platform single node reboot, all pods are restarted which causes significant load and longer than normal pod creation times. This happens because the Container Network Interface (CNI) is not able to process the pod add events quickly enough. The following error message is displayed: timed out waiting for OVS port binding. The OpenShift Container Platform single node instance eventually recovers, though more slowly than expected. This known issue applies to OpenShift Container Platform version 4.8.15 and later. (BZ#1986216)

An error occurs during SNO cluster provisioning where bootkube tries to use oc towards the end of the cluster bootstrap process. The kube API receives a shutdown request and this causes the cluster bootstrap process to fail. (BZ#2010665)

Deploying an OpenShift Container Platform version 4.9 SNO cluster after a successful 4.8 deployment on the same host fails due to a modified boot table entry. (BZ#2011306)

There is an instability issue with the inbox iavf driver that is evident when a DPDK-based workload is deployed in OpenShift Container Platform version 4.8.5. The issue is also apparent when a DPDK workload is deployed on a host running RHEL for Real Time 8. The issue occurs in hosts with Intel XXV710 NICs installed. (BZ#2000180)

A clock jump error occurs in the linuxptp subsystem that is deployed by the PTP Operator. The reported error message is: clock jumped backward or running slower than expected!. The error is encountered in a host with an Intel Columbiaville E810 NIC installed in a OpenShift Container Platform version 4.8 or 4.9 cluster. The error is likely Intel ice driver related, rather than an error in the linuxptp subsystem. (BZ#2013478)

Sometimes Operator installation fails during zero touch provisioning (ZTP) installation of a DU node. The InstallPlan API reports an error. The reported error message is: Bundle unpacking failed. Reason: DeadlineExceeded. The error occurs if the Operator installation job exceeds 600 seconds.

A race condition exists between the SR-IOV Operator and the Machine Config Operator (MCO) which occurs intermittently and manifests itself in different ways during the ZTP installation process for the DU node. The race condition can cause the following errors:

It is not possible to create a macvlan on the physical function (PF) when a virtual function (VF) already exists. This issue affects the Intel E810 NIC. (BZ#2120585)