$ operator-sdk bundle validate .<bundle_dir_or_image> \
--select-optional suite=operatorframework \
--optional-values=k8s-version=1.22
Red Hat OpenShift Container Platform provides developers and IT organizations with a hybrid cloud application platform for deploying both new and existing applications on secure, scalable resources with minimal configuration and management overhead. OpenShift Container Platform supports a wide selection of programming languages and frameworks, such as Java, JavaScript, Python, Ruby, and PHP.
Built on Red Hat Enterprise Linux (RHEL) and Kubernetes, OpenShift Container Platform provides a more secure and scalable multi-tenant operating system for today’s enterprise-class applications, while delivering integrated application runtimes and libraries. OpenShift Container Platform enables organizations to meet security, privacy, compliance, and governance requirements.
OpenShift Container Platform (RHSA-2021:3759) is now available. This release uses Kubernetes 1.22 with CRI-O runtime. New features, changes, and known issues that pertain to OpenShift Container Platform 4.9 are included in this topic.
OpenShift Container Platform 4.9 clusters are available at https://console.redhat.com/openshift. The Red Hat OpenShift Cluster Manager application for OpenShift Container Platform allows you to deploy OpenShift clusters to either on-premise or cloud environments.
OpenShift Container Platform 4.9 is supported on Red Hat Enterprise Linux (RHEL) 7.9 through 8.7 as well as on Red Hat Enterprise Linux CoreOS (RHCOS) 4.9.
You must use RHCOS machines for the control plane, and you can use either RHCOS or Red Hat Enterprise Linux (RHEL) for compute machines.
The scope of support for layered and dependent components of OpenShift Container Platform changes independently of the OpenShift Container Platform version. To determine the current support status and compatibility for an add-on, refer to its release notes. For more information, see the Red Hat OpenShift Container Platform Life Cycle Policy.
This release adds improvements related to the following components and concepts.
Nodes installed with the coreos-installer
program previously retained the installation Ignition config in the /boot/ignition/config.ign
file. Starting with the OpenShift Container Platform 4.9 installation image, that file is removed when the node is provisioned. This change does not affect clusters that were installed on previous OpenShift Container Platform versions because they still use an older bootimage.
OpenShift Container Platform 4.9 introduces support for installing a cluster on Azure Stack Hub using user-provisioned infrastructure.
You can incorporate example Azure Resource Manager (ARM) templates provided by Red Hat to assist in the deployment process, or create your own. You are also free to create the required resources through other methods; the ARM templates are just an example.
See Installing a cluster on Azure Stack Hub using ARM templates for details.
During the upgrade process, nodes in the cluster might become temporarily unavailable. In the case of worker nodes, the machine health check might identify such nodes as unhealthy and reboot them. To avoid rebooting such nodes, OpenShift Container Platform 4.9 introduces the cluster.x-k8s.io/paused=""
annotation to let you pause the MachineHealthCheck
resources before updating the cluster.
For more information, see Pausing a MachineHealthCheck resource.
The OpenShift Container Platform installation program for Microsoft Azure now creates subnets as large as possible within the machine CIDR. This lets the cluster use a machine CIDR that is appropriately sized to accommodate the number of nodes in the cluster.
OpenShift Container Platform 4.9 introduces support for AWS regions in China. You can now install and update OpenShift Container Platform clusters in the cn-north-1
(Beijing) and cn-northwest-1
(Ningxia) regions.
For more information, see Installing a cluster on AWS China.
In OpenShift Container Platform 4.9, you can expand an installer provisioned cluster deployed using the provisioning
network by using Virtual Media on the baremetal
network. You can use this feature when the ProvisioningNetwork
configuration setting is set to Managed
. To use this feature, you must set the virtualMediaViaExternalNetwork
configuration setting to true
in the provisioning
custom resource (CR). You must also edit the machine set to use the API VIP address. See Preparing to deploy with Virtual Media on the baremetal network for details.
OpenShift Container Platform 4.9 uses Kubernetes 1.22, which removed a significant number of deprecated v1beta1
APIs.
OpenShift Container Platform 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.8 to 4.9. This is to help prevent issues after upgrading to OpenShift Container Platform 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.
All OpenShift Container Platform 4.8 clusters require this administrator acknowledgment before they can be upgraded to OpenShift Container Platform 4.9.
For more information, see Preparing to update to OpenShift Container Platform 4.9.
OpenShift Container Platform 4.9 introduces support for installation on Red Hat OpenStack Platform (RHOSP) deployments that rely on PCI passthrough.
OpenShift Container Platform 4.9 supports etcd 3.5. Before you upgrade the cluster, verify that a valid etcd backup exists. An etcd backup ensures that the cluster can be restored if an upgrade failure occurs. In OpenShift Container Platform 4.9, etcd upgrades are automatic. Depending on the cluster’s transition state to version 4.9, an etcd backup might be available. However, verifying that a backup exists before the cluster upgrade starts is recommended.
OpenShift Container Platform 4.9 introduces support for installing a cluster on IBM Cloud® using installer-provisioned infrastructure. The procedure is nearly identical to installer-provisioned infrastructure on bare metal with these differences:
Installer-provisioned installation of OpenShift Container Platform 4.9 on IBM Cloud requires the provisioning
network, IPMI, and PXE boot. Red Hat does not support deployment with Redfish and virtual media on IBM Cloud.
You must create and configure public and private VLANs on the IBM Cloud.
IBM Cloud nodes must be available before starting the installation process. So you must create the IBM Cloud nodes first.
You must prepare the provisioner node.
You must install and configure a DHCP server on the public baremetal
network.
You must configure the install-config.yaml
file so that each node points to the BMC using IPMI, and sets the IPMI privilege level to OPERATOR.
See Deploying installer-provisioned clusters on IBM Cloud for details.
OpenShift Container Platform 4.9 adds BIOS configuration support for worker nodes when deploying installer-provisioned clusters on Fujitsu hardware and using the Fujitsu integrated Remote Management Controller (iRMC). See Configuring BIOS for worker node for details.
With this update, administrators now have the ability to access node logs from the Node page. To review the node logs, you can switch between individual log files and journal log units by clicking the Logs tab.
You now have the ability to filter by node type in the Cluster utilization card on the cluster dashboard. Additional node types will appear in the list when created.
This update adds a User Preferences page for customizing settings, such as default project, perspective, and topology view.
With this update, you can hide default projects
from the Projects dropdown in the web console masthead. You can still toggle to show default projects
before you search and filter.
With this update, you can now add user preferences in the web console. Users can select their default perspective, project, topology, and other preferences.
You can now import a devfile, a Dockerfile, or a builder image through your Git repository to further customize your deployment. You can also edit the file import type and select a different strategy for importing the file.
You can now add tasks in a pipeline using Add task and Quick Search using the updated user interface of the Pipeline builder in the developer console. This enhanced experience allows users to add tasks from the Tekton Hub.
To edit your build configurations, you use the Edit BuildConfig option in the Builds view of the Developer perspective. Users can use a Form view and a YAML view to edit the build configurations.
You can use the context menu in the topology Graph view to add services or create a connection with operator-backed services to the projects.
You can use the +Add actions in the context menu of the topology Graph view to add services or remove a service in the application group.
Initial support for pipeline as code is now available in the Pipelines Repository list view, enabled by the OpenShift Pipelines Operator.
Usability enhancement have been made to the Application Monitoring section in the Observe page of the topology.
With this release, IBM Z and LinuxONE are now compatible with OpenShift Container Platform 4.9. The installation can be performed with z/VM or RHEL KVM. For installation instructions, see the following documentation:
The following new features are supported on IBM Z and LinuxONE with OpenShift Container Platform 4.9:
Helm
Support for multiple network interfaces
Service Binding Operator
The following features are also supported on IBM Z and LinuxONE:
Currently, the following Operators are supported:
Cluster Logging Operator
NFD Operator
OpenShift Elasticsearch Operator
Local Storage Operator
Service Binding Operator
Encrypting data stored in etcd
Multipathing
Persistent storage using iSCSI
Persistent storage using local volumes (Local Storage Operator)
Persistent storage using hostPath
Persistent storage using Fibre Channel
Persistent storage using Raw Block
OVN-Kubernetes
Three-node cluster support
z/VM Emulated FBA devices on SCSI disks
4K FCP block device
These features are available only for OpenShift Container Platform on IBM Z and LinuxONE for 4.9:
HyperPAV enabled on IBM Z and LinuxONE for the virtual machines for FICON attached ECKD storage
Note the following restrictions for OpenShift Container Platform on IBM Z and LinuxONE:
The following OpenShift Container Platform Technology Preview features are unsupported:
Precision Time Protocol (PTP) hardware
The following OpenShift Container Platform features are unsupported:
Automatic repair of damaged machines with machine health checking
CodeReady Containers (CRC)
Controlling overcommit and managing container density on nodes
CSI volume cloning
CSI volume snapshots
FIPS cryptography
Multus CNI plugin
NVMe
OpenShift Metering
OpenShift Virtualization
Tang mode disk encryption during OpenShift Container Platform deployment
Worker nodes must run Red Hat Enterprise Linux CoreOS (RHCOS)
Persistent shared storage must be provisioned by using either NFS or other supported storage protocols
Persistent non-shared storage must be provisioned using local storage, like iSCSI, FC, or using LSO with DASD, FCP, or EDEV/FBA
With this release, IBM Power Systems are now compatible with OpenShift Container Platform 4.9. For installation instructions, see the following documentation:
The following new features are supported on IBM Power Systems with OpenShift Container Platform 4.9:
Helm
Support for Power10
Support for multiple network interfaces
Service Binding Operator
The following features are also supported on IBM Power Systems:
Currently, the following Operators are supported:
Cluster Logging Operator
NFD Operator
OpenShift Elasticsearch Operator
Local Storage Operator
SR-IOV Network Operator
Service Binding Operator
Multipathing
Persistent storage using iSCSI
Persistent storage using local volumes (Local Storage Operator)
Persistent storage using hostPath
Persistent storage using Fibre Channel
Persistent storage using Raw Block
OVN-Kubernetes
4K Disk Support
NVMe
Encrypting data stored in etcd
Three-node cluster support
Multus SR-IOV
Note the following restrictions for OpenShift Container Platform on IBM Power Systems:
The following OpenShift Container Platform Technology Preview features are unsupported:
Precision Time Protocol (PTP) hardware
The following OpenShift Container Platform features are unsupported:
Automatic repair of damaged machines with machine health checking
CodeReady Containers (CRC)
Controlling overcommit and managing container density on nodes
FIPS cryptography
OpenShift Metering
OpenShift Virtualization
Tang mode disk encryption during OpenShift Container Platform deployment
Worker nodes must run Red Hat Enterprise Linux CoreOS (RHCOS)
Persistent storage must be of the Filesystem type that uses local volumes, Network File System (NFS), or Container Storage Interface (CSI)
You now have more fine-grained control over the audit logging level for OpenShift Container Platform. You can use custom rules to specify a different audit policy profile (Default
, WriteRequestBodies
, AllRequestBodies
, or None
) for different groups.
For more information, see Configuring the audit log policy with custom rules.
You can now disable audit logging for OpenShift Container Platform by using the None
audit policy profile.
It is not recommended to disable audit logging unless you are fully aware of the risks of not logging data that can be beneficial when troubleshooting issues. If you disable audit logging and a support situation arises, you might need to enable audit logging and reproduce the issue in order to troubleshoot properly. |
For more information, see Disabling audit logging.
You can now customize the URL for the internal OAuth server. For more information, see Customizing the internal OAuth server URL.
OpenShift Container Platform 4.9 provides new procedures for ongoing maintenance of NBDE-configured systems. NBDE allows you to encrypt root volumes of hard drives on physical and virtual machines without having to manually enter a password when restarting machines. For more information, see About disk encryption technology.
OpenShift Container Platform 4.9 introduces the following updates to PTP:
New ptp4lConf
field
New option to configure linuxptp
services as a boundary clock
For more information, see Configuring linuxptp services as boundary clock.
Fast event notifications for PTP events are now available for bare-metal clusters. The PTP Operator generates event notifications for every configured PTP-capable network interface. Events are made available through a REST API for applications running on the same node. Fast event notifications are transported by an Advanced Message Queuing Protocol (AMQP) message bus provided by the AMQ Interconnect Operator.
For more information, see About PTP and clock synchronization error events.
The egress IP feature of OVN-Kubernetes now balances network traffic approximately equally across nodes for a given namespace, if that namespace is assigned multiple egress IP addresses. Each IP address must reside on a different node. For more information, refer to Configuring egress IPs for a project for OVN-Kubernetes.
The containerized Data Plane Development Kit (DPDK) is now GA in OpenShift Container Platform 4.9. For more information, see Using virtual functions (VFs) with DPDK and RDMA modes.
SR-IOV now supports vhost-net for use with Fast Datapath DPDK applications on Intel and Mellanox NICs. You can enable this feature by configuring the SriovNetworkNodePolicy
resource. For more information, see SR-IOV network node configuration object.
Single node clusters support SR-IOV hardware and the SR-IOV Network Operator. Be aware that configuring an SR-IOV network device causes the single node to reboot and that you must configure the disableDrain
field for the Operator. For more information, see Configuring the SR-IOV Network Operator.
OpenShift Container Platform 4.9 adds support for additional Broadcom and Intel hardware.
Broadcom BCM57414 and BCM57508
For more information, see the supported devices.
This release introduces the MetalLB Operator. After installing and configuring the MetalLB Operator, you can deploy MetalLB to provide a native load balancer implementation for services on bare-metal clusters. Other on-premise infrastructures that are like bare metal can also benefit.
The Operator introduces a custom resource, AddressPool
. You configure address pools with ranges of IP addresses that MetalLB can assign to services. When you add a service of type LoadBalancer
, MetalLB assigns an IP address from a pool.
For this release, Red Hat only supports using MetalLB in layer 2 mode.
For more information, see About MetalLB and the MetalLB Operator.
The CNI VRF plugin was previously introduced as a Technology Preview feature in OpenShift Container Platform 4.7 and is now generally available in OpenShift Container Platform 4.9.
For more information, see Assigning a secondary network to a VRF.
This release introduces six timeout configurations for the Ingress Controller tuningOptions
parameter:
clientTimeout
specifies how long a connection is held open while waiting for a client response.
serverFinTimeout
specifies how long a connection is held open while waiting for the server response to the client that is closing the connection.
serverTimeout
specifies how long a connection is held open while waiting for a server response.
clientFinTimeout
specifies how long a connection is held open while waiting for the client response to the server closing the connection.
tlsInspectDelay
specifies how long the router can hold data to find a matching route.
tunnelTimeout
specifies how long a tunnel connection, including WebSocket connections, remains open while the tunnel is idle.
For more information, see Ingress controller configuration parameters.
You can now configure the Ingress Controller to enable mutual TLS (mTLS) authentication by setting spec.clientTLS
. The clientTLS
field specifies configuration for the Ingress Controller to verify client certificates.
For more information, see Configuring Mutual TLS Authentication.
Cluster administrators can specify a custom HTTP error code response page for either 503, 404, or both error pages.
For more information, see Customizing HAProxy error code response pages.
In OpenShift Container Platform 4.9, the provisioningNetworkInterface
configuration setting for installer-provisioned clusters is optional. The provisioningNetworkInterface
configuration setting identifies the NIC name used for the provisioning
network. In OpenShift Container Platform 4.9, you can alternatively specify the bootMACAddress
configuration setting in the install-config.yml
file, which enables Ironic to identify the IP address for the NIC connected to the provisioning
network and bind to it. You can also omit the provisioningInterface
configuration setting in the provisioning custom resource so that the provisioning custom resource uses the bootMACAddress
configuration setting instead.
In OpenShift Container Platform 4.9, you can now change the DNS Operator managementState
. The managementState
of the DNS Operator is set to Managed
by default, which means that the DNS Operator is actively managing its resources. You can change it to Unmanaged
, which means the DNS Operator is not managing its resources.
The following are use cases for changing the DNS Operator managementState
:
You are a developer and want to test a configuration change to see if it fixes an issue in CoreDNS. You can stop the DNS Operator from overwriting the change by setting the managementState
to Unmanaged
.
You are a cluster administrator and have reported an issue with CoreDNS, but need to apply a workaround until the issue is fixed. You can set the managementState
field of the DNS Operator to Unmanaged
to apply the workaround.
For more information, see Changing the DNS Operator managementState.
For clusters that run on RHOSP, you can now configure Octavia for load balancing as a cloud provider option.
For more information, see Setting cloud provider options.
This release adds Ingress Controller support for TLS 1.3 and the Modern
profile in HAProxy.
For more information, see Ingress Controller TLS security profiles.
Cluster administrators can configure HTTP Strict Transport Security (HSTS) verification on a per-domain basis with the addition of an admission plugin for the router, called route.openshift.io/RequiredRouteAnnotations
. If a cluster administrator configures this plugin to enforce HSTS, then any newly created route must be configured with a compliant HSTS Policy, which is verified against the global setting on the cluster Ingress configuration, called ingresses.config.openshift.io/cluster
.
For more information, see HTTP Strict Transport Security.
In OpenShift Container Platform 4.9 you can now configure the Ingress Controller to log or ignore empty requests by setting the logEmptyRequests
and HTTPEmptyRequestsPolicy
fields.
For more information, see Ingress controller configuration parameters.
OpenShift Container Platform is capable of provisioning persistent volumes (PVs) using the Container Storage Interface (CSI) driver for AWS Elastic Block Store (EBS). This feature was previously introduced as a Technology Preview feature in OpenShift Container Platform 4.5 and is now generally available and enabled by default in OpenShift Container Platform 4.9.
For more information, see AWS EBS CSI Driver Operator.
OpenShift Container Platform is capable of provisioning PVs using the CSI driver for Azure Stack Hub Storage. Azure Stack Hub, which is part of the Azure Stack portfolio, allows you to run apps in an on-premises environment and deliver Azure services in your datacenter. The Azure Stack Hub CSI Driver Operator that manages this driver is new for 4.9 and generally available.
For more information, see Azure Stack Hub CSI Driver Operator.
OpenShift Container Platform is capable of provisioning PVs using the CSI driver for AWS Elastic File Service (EFS). The AWS EFS CSI Driver Operator that manages this driver is in Technology Preview.
For more information, see AWS EFS CSI Driver Operator.
Starting with OpenShift Container Platform 4.8, automatic migration for in-tree volume plugins to their equivalent CSI drivers became available as a Technology Preview feature. This feature now supports automatic migration from Google Compute Engine Persistent Disk (GCE PD) in-tree plugin to the Google Cloud Platform (GCP) Persistent Disk CSI driver.
For more information, see CSI Automatic Migration.
Starting with OpenShift Container Platform 4.8, automatic migration for in-tree volume plugins to their equivalent CSI drivers became available as a Technology Preview feature. This feature now supports automatic migration from the Azure Disk in-tree plugin to the Azure Disk CSI driver.
For more information, see CSI Automatic Migration.
The vSphere CSI Operator Driver storage class now uses vSphere’s storage policy. OpenShift Container Platform automatically creates a storage policy that targets datastore configured in cloud configuration.
For more information, see VMWare vSphere CSI Driver Operator.
OpenShift Container Platform 4.9 provides the following new metrics for the Local Storage Operator:
lso_discovery_disk_count
: total number of discovered devices on each node
lso_lvset_provisioned_PV_count
: total number of PVs created by LocalVolumeSet
objects
lso_lvset_unmatched_disk_count
: total number of disks that Local Storage Operator did not select for provisioning because of mismatching criteria
lso_lvset_orphaned_symlink_count
: number of devices with PVs that no longer match LocalVolumeSet
object criteria
lso_lv_orphaned_symlink_count
: number of devices with PVs that no longer match LocalVolume
object criteria
lso_lv_provisioned_PV_count
: total number of provisioned PVs for LocalVolume
For more information, see Persistent storage using local volumes.
OpenShift Container Platform 4.9 adds resizing capability to the oVirt CSI Driver, which allows users to increase the size of their existing persistent volume claims (PVCs). Prior to this feature, users had to create new PVCs with the increased size, and move all of the content from the old persistent volume (PV) to the new PV, which could result in data loss. Now, users can edit the existing PVC and the oVirt CSI Driver will resize the underlying oVirt disk.
In OpenShift Container Platform 4.9, the integrated Image Registry uses Azure Blob Storage for clusters installed on Microsoft Azure Stack Hub using user-provisioned infrastructure.
See Installing a cluster on Azure Stack Hub using ARM templates for details.
The following new features and enhancements relate to running Operators with Operator Lifecycle Manager (OLM).
Starting in OpenShift Container Platform 4.9, Operator Lifecycle Manager (OLM) supports Kubernetes 1.22. As a result, a significant number of v1beta1
APIs have been removed and updated to v1
. Operators that depend on the removed v1beta1
APIs will not run on OpenShift Container Platform 4.9. Cluster administrators should upgrade their installed Operators to the latest
channel before upgrading a cluster to OpenShift Container Platform 4.9.
Kubernetes 1.22 introduces several notable changes to |
File-based catalogs are the latest iteration of the catalog format in Operator Lifecycle Manager (OLM). The format is a plain text-based (JSON or YAML) and declarative config evolution of the earlier, and now deprecated, SQLite database format, and it is fully backwards compatible. The goal of this format is to enable Operator catalog editing, composability, and extensibility.
For more information about the file-based catalog specification, see Operator Framework packaging format.
For instructions about creating file-based catalogs by using the opm
CLI, see Managing custom catalogs.
Operator Lifecycle Manager (OLM) is now available on Single Node OpenShift (SNO) clusters, enabling self-service Operator installations.
Because administrators should not require an understanding of the interaction process between the various low-level APIs or access to the Operator Lifecycle Manager (OLM) pod logs to successfully debug such issues, OpenShift Container Platform 4.9 introduces the following enhancements in OLM to provide administrators with more comprehensible error reporting and messages:
Previously, if a namespace contained multiple Operator groups or could not find a service account, the status of the Operator group would not report an error. With this enhancement, these scenarios now update the status condition of the Operator group to report an error.
To avoid cluster upgrades potentially leaving Operator installations in an unsupported state or without a continued update path, you can enable automatically changing your Operator catalog’s index image version as part of cluster upgrades.
Set the olm.catalogImageTemplate
annotation to your catalog image name and use one or more of the Kubernetes cluster version variables when constructing the template for the image tag.
For more information, see Image template for custom catalog sources.
The following new features and enhancements relate to developing Operators with the Operator SDK.
An OpenShift Container Platform cluster can be configured in high-availability (HA) mode, which uses multiple nodes, or in non-HA mode, which uses a single node. A single node cluster, also known as Single Node OpenShift (SNO), is likely to have more conservative resource constraints. Therefore, it is important that Operators installed on a single node cluster can adjust accordingly and still run well.
By accessing the cluster high-availability mode API provided in OpenShift Container Platform, Operator authors can use the Operator SDK to enable their Operator to detect a cluster’s infrastructure topology, either HA or non-HA mode. Custom Operator logic can be developed that uses the detected cluster topology to automatically switch the resource requirements, both for the Operator and for any Operands or workloads it manages, to a profile that best fits the topology.
For more information, see High-availability or single node cluster detection and support.
Operator authors can now develop Operators that support network proxies. Operators with proxy support inspect the Operator deployment for environment variables and pass the variables on to the required Operands. Cluster administrators configure proxy support for the environment variables that are handled by Operator Lifecycle Manager (OLM). For more information, see the Operator SDK tutorials for developing Operators using Go, Ansible, and Helm.
You can now check bundle manifests for APIs removed from Kubernetes 1.22 by using the Operator Framework suite of tests with the bundle validate
subcommand.
For example:
$ operator-sdk bundle validate .<bundle_dir_or_image> \
--select-optional suite=operatorframework \
--optional-values=k8s-version=1.22
If your bundle manifest includes APIs removed from Kubernetes 1.22, the command displays a warning message. The warning message indicates which APIs you need to migrate and links to the Kubernetes API migration guide.
See the table of beta APIs removed from Kubernetes 1.22 and the Operator SDK CLI reference for more information.
As a developer using OpenShift Container Platform for builds, with this update, you can use the following new capabilities:
You can mount build volumes to give running builds access to information that you do not want to persist in the output container image. Build volumes can provide sensitive information, such as repository credentials, which the build environment or configuration only needs at build-time. Build volumes are different from build inputs, whose data can persist in the output container image.
You can configure image changes to trigger builds based on information recorded in the BuildConfig status. This way, you can use ImageChange
triggers with builds in a GitOps workflow.
This release introduces support for using wildcard domains as registry sources in your image registry settings. With a wildcard domain, such as *.example.com
, you can set your cluster to push and pull images from multiple subdomains without having to manually enter each one. For more information, see Image controller configuration parameters.
Starting in OpenShift Container Platform 4.9, you can now use Red Hat Enterprise Linux (RHEL) 8.4 for compute machines. Previously, RHEL 8 was not supported for compute machines.
You cannot upgrade RHEL 7 compute machines to RHEL 8. You must deploy new RHEL 8 hosts, and the old RHEL 7 hosts should be removed.
Scheduling pods using a scheduler profile is now generally available. This is a replacement for configuring a scheduler policy. The following scheduler profiles are available:
LowNodeUtilization
: This profile attempts to spread pods evenly across nodes to get low resource usage per node.
HighNodeUtilization
: This profile attempts to place as many pods as possible onto as few nodes as possible, to minimize node count with high usage per node.
NoScoring
: This is a low-latency profile that strives for the quickest scheduling cycle by disabling all score plugins. This might sacrifice better scheduling decisions for faster ones.
For more information, see Scheduling pods using a scheduler profile.
The following descheduler profiles are now available:
SoftTopologyAndDuplicates
: This profile is the same as TopologyAndDuplicates
, except that pods with soft topology constraints, such as whenUnsatisfiable: ScheduleAnyway
, are also considered for eviction.
EvictPodsWithLocalStorage
: This profile allows pods with local storage to be eligible for eviction.
EvictPodsWithPVC
: This profile allows pods with persistent volume claims to be eligible for eviction.
You can also customize the pod lifetime value for the LifecycleAndUtilization
profile.
For more information, see Evicting pods using the descheduler.
When configuring the docker/config.json
file to allow pods to pull images from private registries, you can now list specific repositories in the same registry, each with credentials specific to that registry path. Previously, you could list only one repository from a given registry. You can also now define a registry with a specific namespace.
Node-related metrics and alerts have been enhanced to give you an earlier indication of when the stability of a node is compromised.
The Poison Pill Operator now provides enhanced remediation with the use of watchdog devices. For more information, see About watchdog devices.
In OpenShift Container Platform 4.7, Cluster Logging became Red Hat OpenShift Logging. For more information, see Release notes for Red Hat OpenShift Logging.
The monitoring stack for this release includes the following new and modified features.
Updates to versions of monitoring stack components and dependencies include the following:
Prometheus to 2.29.2
The Prometheus Operator to 0.49.0
The Prometheus Adapter to 0.9.0
Alertmanager to 0.22.2
Thanos to 0.22.0
New
HighlyAvailableWorkloadIncorrectlySpread
informs you about a potential problem when two instances of a highly available monitoring component are running on the same node and have persistent volumes attached.
NodeFileDescriptorLimit
triggers an alert when a node kernel is running out of available file descriptors. A warning level alert fires at greater than 70% usage, and a critical level alert fires at greater than 90% usage.
PrometheusLabelLimitHit
detects when a target exceeds the defined label limits.
PrometheusTargetSyncFailure
detects when Prometheus fails to synchronize targets.
All critical alerting rules contain links to runbooks.
Enhanced
AlertmanagerReceiversNotConfigured
and KubePodCrashLooping
now contain fewer false positives.
KubeCPUOvercommit
and KubeMemoryOvercommit
are now more robust in non-homogeneous environments.
The for
duration setting of the NodeFilesystemAlmostOutOfSpace
alerting rule has changed from one hour to 30 minutes so that the system more quickly detects when disk space runs low.
KubeDeploymentReplicasMismatch
now fires as expected. In previous versions, this alert did not fire.
The following alerts now contain a namespace
label:
AlertmanagerReceiversNotConfigured
KubeClientErrors
KubeCPUOvercommit
KubeletDown
KubeMemoryOvercommit
MultipleContainersOOMKilled
ThanosQueryGrpcClientErrorRate
ThanosQueryGrpcServerErrorRate
ThanosQueryHighDNSFailures
ThanosQueryHttpRequestQueryErrorRateHigh
ThanosQueryHttpRequestQueryRangeErrorRateHigh
ThanosSidecarPrometheusDown
Watchdog
Red Hat does not guarantee backward compatibility for metrics, recording rules, or alerting rules. |
You can add and configure additional external Alertmanagers for both platform and user-defined project monitoring stacks.
You can disable the local Alertmanager instance.
With the new monitoring-alertmanager-edit
user role, non-administrator users can create and silence alerts for default platform monitoring.
To allow these users to create and silence alerts, you must assign them the new monitoring-alertmanager-edit
role in addition to the cluster-monitoring-view
role.
With this release, the |
You can enable and configure remote write storage for both platform monitoring and user-defined projects in Prometheus. This feature enables you to send ingested metrics to long-term storage.
To reduce the overall memory consumption of Prometheus, the following cAdvisor metrics with both an empty pod
and namespace
label have been dropped:
container_fs_.*
container_spec_.*
container_blkio_device_usage_total
container_file_descriptors
container_sockets
container_threads_max
container_threads
container_start_time_seconds
container_last_seen
When persistent storage is not configured for platform monitoring, upgrades and cluster disruptions can lead to data loss. A warning message has been added to the Degraded
condition when the system detects that persistent storage is not configured for platform monitoring.
You can exclude individual user-defined projects from the openshift-user-workload-monitoring
project by adding the openshift.io/user-monitoring: "false"
label to them.
You can configure an enforcedTargetLimit
parameter for the openshift-user-workload-monitoring
project to set an overall limit on the number of targets scraped.
The link to the third-party Prometheus UI is removed from the Observe → Metrics page in the OpenShift Container Platform web console. You can still access the route to the Prometheus UI in the web console in the Administrator perspective by navigating to the Networking → Routes page in the openshift-monitoring
project.
You can now use the Special Resource Operator (SRO) to help manage the deployment of kernel modules and drivers on an existing OpenShift Container Platform cluster. This is currently a Technology Preview feature.
For more information, see About the Special Resource Operator.
The Memory Manager, a kubelet subcomponent configured by the Performance Addon Operator, is now enabled by default for all pods running on the node that is configured with one of the following Topology Manager policies:
single-numa-node
restricted
For more information, see Topology Manager policies.
OpenShift Container Platform 4.9 introduces two additional tools to measure system latency:
hwladetect
measures the baseline that the bare hardware can achieve
cyclictest
schedules a repeated timer after hwlatdetect
passes validation and measures the difference between the desired and the actual trigger times
For more information, see Running the latency tests.
Updated guidance around cluster maximums for OpenShift Container Platform 4.9 is now available.
No large scale testing for performance against OVN-Kubernetes testing was executed for this release. |
Use the OpenShift Container Platform Limit Calculator to estimate cluster limits for your environment.
OpenShift Container Platform 4.9 supports zero touch provisioning (ZTP), which allows you to provision new edge sites with declarative configurations of bare metal equipment at remote sites. ZTP uses the GitOps deployment set of practices for infrastructure deployment. GitOps achieves these tasks using declarative specifications stored in Git repositories, such as YAML files and other defined patterns, to provide a framework for deploying the infrastructure. The declarative output is leveraged by the Open Cluster Manager (OCM) for multisite deployment. For more information, see Provisioning edge sites at scale.
In OpenShift Container Platform 4.9, Insights Operator can import RHEL Simple Content Access (SCA) certificates from Red Hat OpenShift Cluster Manager.
For more information, see Importing RHEL Simple Content Access certificates with Insights Operator.
In OpenShift Container Platform 4.9, the Insights Operator collects the following additional information:
All of the MachineConfig
resource definitions from a cluster.
The names of the PodSecurityPolicies
installed in a cluster.
If installed, the ClusterLogging
resource definition.
If the SamplesImagestreamImportFailing
alert is firing, then the ImageStream
definitions and the last 100 lines of container logs from the openshift-cluster-samples-operator
namespace.
With this additional information, Red Hat can provide improved remediation steps in Insights Advisor.
With this release, installations on Microsoft Azure Stack Hub can be performed by configuring the Cloud Credential Operator (CCO) in manual mode.
For more information, see Using manual mode.
To review OpenShift sandboxed containers new features, bug fixes, known issues, and asynchronous errata updates, see OpenShift sandboxed containers 1.1 release notes.
OpenShift Container Platform 4.9 introduces the following notable technical changes.
In OpenShift Container Platform 4.9, etcd data is automatically defragmented by the etcd Operator.
Previously, on Red Hat OpenStack Platform (RHOSP) deployments, opening traffic on NodePorts was constrained to the CIDR of the node’s subnet. In order to support LoadBalancer services using the Octavia Open Virtual Network (OVN) provider, the security group rules that allow NodePort traffic to master and worker nodes are now changed to open 0.0.0.0/0
.
The Red Hat OpenStack Platform (RHOSP) cloud provider LoadBalancer configuration now defaults to use-octavia=True
. An exception to this rule is a deployment with Kuryr, in which case use-octavia
is set to false
, because Kuryr handles LoadBalancer services on its own.
The OpenShift Container Platform Ingress Controller is upgraded to HAProxy version 2.2.15.
In OpenShift Container Platform 4.9, CoreDNS uses version 1.8.4, which includes bug fixes.
The Kubernetes controller manager that manages cloud provider deployments does not include support for Azure Stack Hub as a provider. Because using cloud controller managers is the preferred method for interacting with underlying cloud platforms, there is no plan to add this support. As a result, the Azure Stack Hub implementation in OpenShift Container Platform uses cloud controller managers.
In addition, this release supports using cloud controller managers for Amazon Web Services (AWS), Microsoft Azure, and Red Hat OpenStack Platform (RHOSP) as a Technology Preview. Any new cloud platform support that is added to OpenShift Container Platform will also use cloud controller managers.
To learn more about the cloud controller manager, see the Kubernetes documentation on this component.
To manage the cloud controller manager and cloud node manager deployments and lifecycles, this release introduces the Cluster Cloud Controller Manager Operator.
For more information, see the Cluster Cloud Controller Manager Operator entry in the Red Hat Operators reference.
With OpenShift Container Platform 4.9, a new process to perform a canary rollout update has been introduced. For a detailed overview of this process, see Performing a canary rollout update.
Operator Lifecycle Manager (OLM) now compresses Operator bundles with large amounts of metadata, such as large custom resource definition (CRD) manifests, to stay below the 1 MB limit set by etcd.
Operator Lifecycle Management (OLM) catalog pods are now more efficient and use less RAM.
Operators that ship with OpenShift Container Platform "Extras" advisories, such as RHBA-2021:3760, are published in Red Hat-provided catalogs and run on Operator Lifecyle Manager (OLM). Starting with OpenShift Container Platform 4.9, these Operators are now included in a stable
update channel in addition to the version-specific 4.9
channel.
For OpenShift Container Platform 4.9 and future releases, stable
will be the default channel for these Operators. Cluster administrators should use the stable
channel so that changing update channels for these Operators in OLM is no longer necessary with future cluster upgrades.
For more information about OLM-based Operators, see Red Hat-provided Operator catalogs and Understanding OperatorHub. For more information about update channels in OLM, see Upgrading installed Operators.
OpenShift Container Platform 4.9 supports Operator SDK v1.10.1. See Installing the Operator SDK CLI to install or update to this latest version.
Operator SDK v1.10.1 supports Kubernetes 1.21. |
If you have any Operator projects that were previously created or maintained with Operator SDK v1.8.0, see Upgrading projects for newer Operator SDK versions to ensure your projects are upgraded to maintain compatibility with Operator SDK v1.10.1.
Some features available in previous releases have been deprecated or removed.
Deprecated functionality is still included in OpenShift Container Platform and continues to be supported; however, it will be removed in a future release of this product and is not recommended for new deployments. For the most recent list of major functionality deprecated and removed within OpenShift Container Platform 4.9, refer to the table below. Additional details for more fine-grained functionality that has been deprecated and removed are listed after the table.
In the table, features are marked with the following statuses:
GA: General Availability
TP: Technology Preview
DEP: Deprecated
REM: Removed
Feature | OCP 4.7 | OCP 4.8 | OCP 4.9 |
---|---|---|---|
Package manifest format (Operator Framework) |
DEP |
REM |
REM |
SQLite database format for Operator catalogs |
GA |
GA |
DEP |
|
DEP |
REM |
REM |
|
DEP |
REM |
REM |
v1beta1 CRDs |
DEP |
DEP |
REM |
Docker Registry v1 API |
DEP |
DEP |
REM |
Metering Operator |
DEP |
DEP |
REM |
Scheduler policy |
DEP |
DEP |
DEP |
|
DEP |
DEP |
DEP |
|
DEP |
DEP |
DEP |
Use of |
DEP |
DEP |
REM |
Use of |
DEP |
DEP |
REM |
Cluster Loader |
GA |
DEP |
DEP |
Bring your own RHEL 7 compute machines |
DEP |
DEP |
DEP |
|
GA |
DEP |
REM |
Jenkins Operator |
TP |
DEP |
DEP |
HPA custom metrics adapter based on Prometheus |
TP |
REM |
REM |
vSphere 6.7 Update 2 or earlier and virtual hardware version 13 |
GA |
GA |
DEP |
The |
DEP |
DEP |
DEP |
Minting credentials for Microsoft Azure clusters |
GA |
GA |
The SQLite database format used by Operator Lifecycle Manager (OLM) for catalogs and index images has been deprecated, including the related opm
CLI commands. Cluster administrators and catalog maintainers are encouraged to familiarize themselves with the new file-based catalog format introduced in OpenShift Container Platform 4.9 and begin migrating catalog workflows.
The default Red Hat-provided Operator catalogs for OpenShift Container Platform 4.6 and later are currently still shipped in the SQLite database format. |
Installing a cluster on VMware vSphere version 6.7 Update 2 or earlier and virtual hardware version 13 is now deprecated. Support for these versions will end in a future version of OpenShift Container Platform.
Hardware version 15 is now the default for vSphere virtual machines in OpenShift Container Platform. Hardware version 15 will be the only supported version in a future version of OpenShift Container Platform.
Kubernetes 1.22 removed the following deprecated v1beta1
APIs. Migrate manifests and API clients to use the v1
API version. For more information about migrating removed APIs, see the Kubernetes documentation.
Resource | API | Notable changes |
---|---|---|
|
|
No |
|
|
|
|
|
No |
|
|
No |
|
|
No |
|
|
No |
|
|
|
|
|
|
|
|
|
|
|
No |
|
|
No |
|
|
|
|
|
|
|
|
No |
|
|
No |
|
|
No |
|
|
|
|
|
No |
|
|
|
|
|
No |
|
|
|
|
|
No |
The deprecated v1beta1
API for the descheduler has been removed in OpenShift Container Platform 4.9. Migrate any resources using the descheduler v1beta1
API version to v1
.
The deprecated dhclient
binary has been removed from RHCOS. Starting with OpenShift Container Platform 4.6, RHCOS switched to using NetworkManager
in the initramfs
to configure networking during early boot. Use the NetworkManager
internal DHCP client for networking configuration instead. See BZ#1908462 for more information.
The current release stops updating the buildConfig.spec.triggers[i].imageChange.lastTriggeredImageID
field when the ImageStreamTag
referenced by buildConfig.spec.triggers[i].imageChage
points to a new image. Instead, this release updates the buildConfig.status.imageChangeTriggers[i].lastTriggeredImageID
field.
Additionally, the Build Image Change Trigger controller ignores the buildConfig.spec.triggers[i].imageChange.lastTriggeredImageID
field.
Now, the Build Image Change Trigger controller starts a build based on the buildConfig.status.imageChangeTriggers[i].lastTriggeredImageID
field and how it compares to the image ID now referenced by the ImageStreamTag
referenced in the buildConfig.spec.triggers[i].imageChange
.
Therefore, update scripts and jobs that inspect buildConfig.spec.triggers[i].imageChange.lastTriggeredImageID
accordingly. (BUILD-190)
Support for using v1
without a group for apiVersion
for OpenShift Container Platform resources has been removed. Every resource that includes *.openshift.io
must match the apiVersion
value found in the API index.
Starting with OpenShift Container Platform 4.9.24, support for using the Cloud Credential Operator (CCO) in mint mode on Microsoft Azure clusters has been removed from OpenShift Container Platform 4.9. This change is due to the planned retirement of the Azure AD Graph API by Microsoft on 30 June 2022 and is being backported to all supported versions of OpenShift Container Platform in z-stream updates.
For previously installed Azure clusters that use mint mode, the CCO attempts to update existing secrets. If a secret contains the credentials of previously minted app registration service principals, it is updated with the contents of the secret in kube-system/azure-credentials
. This behavior is similar to passthrough mode.
For clusters with the credentials mode set to its default value of ""
, the updated CCO automatically changes from operating in mint mode to operating in passthrough mode. If your cluster has the credentials mode explicitly set to mint mode ("Mint"
), you must change the value to ""
or "Passthrough"
.
In addition to the |
While the Azure AD Graph API is still available, the CCO in upgraded versions of OpenShift Container Platform attempts to clean up previously minted app registration service principals. Upgrading your cluster before the Azure AD Graph API is retired might avoid the need to clean up resources manually.
If the cluster is upgraded to a version of OpenShift Container Platform that no longer supports mint mode after the Azure AD Graph API is retired, the CCO sets an OrphanedCloudResource
condition on the associated CredentialsRequest
but does not treat the error as fatal. The condition includes a message similar to unable to clean up App Registration / Service Principal: <app_registration_name>
. Cleanup after the Azure AD Graph API is retired requires manual intervention using the Azure CLI tool or the Azure web console to remove any remaining app registration service principals.
To clean up resources manually, you must find and delete the affected resources.
Using the Azure CLI tool, filter the app registration service principals that use the <app_registration_name>
from an OrphanedCloudResource
condition message by running the following command:
$ az ad app list --filter "displayname eq '<app_registration_name>'" --query '[].objectId'
[
"038c2538-7c40-49f5-abe5-f59c59c29244"
]
Delete the app registration service principal by running the following command:
$ az ad app delete --id 038c2538-7c40-49f5-abe5-f59c59c29244
After cleaning up resources manually, the |
Previously, encryption conditions could remain indefinitely and be reported as a degraded condition for some Operators. Stale encryption conditions are now cleared properly and no longer improperly reported. (BZ#1974520)
Previously, the CA for API server client certificates was rotated early in the lifetime of a cluster, which prevented the Authentication Operator from creating a certificate signing request (CSR) because a previous CSR with the same name still existed. The Kubernetes API server was unable to authenticate itself to the OAuth API server when sending TokenReview
requests, which caused authentication to fail. Generated names are now used when creating CSRs by the Authentication Operator, so an early rotation of the CA for API server client certificates no longer causes authentication failures.
(BZ#1978193)
Previously, metal3 pods could not download an Red Hat Enterprise Linux CoreOS (RHCOS) image due to the sequencing of creating initContainers. This issue is fixed by reordering the creation of the initContainers, so that the metal-static-ip-set
initContainer is created before the metal3-machine-os-downloader
initContainer. The RHCOS image now downloads as expected.
(BZ#1973724)
Previously, when using installer-provisioned installation on bare metal with a host configured to use idrac-virtualmedia
, the bios_interface
for that host got set to idrac-wsman
by default. This resulted in the BIOS settings being unavailable and an exception occurring. This issue is fixed by using idrac-redfish
for the default bios_interface
when using idrac-virtualmedia
.
(BZ#1928816)
Previously, in UEFI mode, the ironic-python-agent
created a UEFI bootloader entry after downloading the RHCOS image. When using an RHCOS image based on RHEL 8.4, the image could fail to boot using this entry and output a BIOS error screen. This is fixed by the ironic-python-agent
configuring the boot entry based on a CSV file located in the image, instead of using a fixed boot entry. The image boots properly without error.
(BZ#1966129)
Previously, if provisioningHostIP
had been set in install-config
, it was assigned to the metal3 pod, even in cases where the provisioning network had been disabled. This has been fixed.
(BZ#1972753)
Previously, the assisted installer could not provision Supermicro X11/X12-based systems because of a mismatch in the sushy resource library. The mismatch resulted in an installation issue by being unable to attach virtual media to the Inserted
and WriteProtected
attributes, and not being allowed in the VirtualMedia.InsertMedia
request body. This issue is fixed by modifying the sushy resource library, and adding a condition to stop sending these optional attributes when not strictly required, thus allowing the installation to progress past this point.
(BZ#1986238)
Previously, some error types in the provisioned state caused the host to be deprovisioned. This occurred after a restart of the metal3 pod if the image provisioned to a bare metal host became unavailable. In this case, the host would enter the deprovisioning state. This issue is fixed by modifying the action of the error in the provisioned state so that if the image becomes unavailable, the error will be reported but deprovisioning will not be initiated. (BZ#1972374)
In OpenShift Container Platform and later, the fix for bug BZ#1884270 incorrectly pruned SSH protocol URLs in an attempt to provide SCP-styled URL capabilities. This error caused the oc new-build
command not to pick an automatic source clone secret: the build could not use the build.openshift.io/sbuild.openshift.io/source-secret-match-uri-1ource-secret-match-uri-1
annotation to map SSH keys with the associated secrets, and therefore could not perform git cloning. This update reverts the changes from BZ#1884270 so that builds can use the annotation and perform git cloning.
Before this update, various allowed and block registry configuration options of the cluster image configuration could block the Cluster Samples Operator from creating image streams. When that happened, the samples operator marked itself as degraded
, which impacted the general OpenShift Container Platform install and upgrade status.
The Cluster Samples Operator can bootstrap itself as removed
in a variety of circumstances. With this update, these circumstances include when the image controller configuration parameters prevent the creation of image streams by using the default image registry or by using the image registry specified by the samplesRegistry
setting. The Operator status also indicates when the cluster image configuration prevents the creation of the sample image streams.
Previously, when a root volume was created for a new server, and that server was not successfully created, the automatic deletion for the volume was not triggered because there was no deletion of a server associated with the volume. In some situations, this led to the creation of many extra volumes, and caused errors if the volume quota was reached. With this release, newly created root volumes are deleted when server creation call fails. (