Preparing to upgrade ROSA to 4.9 | Upgrading | Red Hat OpenShift Service on AWS

Requirements for upgrading to OpenShift 4.9
- Administrator acknowledgment when upgrading to OpenShift 4.9
- Removed Kubernetes APIs
Evaluating your cluster for removed APIs
Migrating instances of removed APIs

Upgrading your Red Hat OpenShift Service on AWS clusters to OpenShift 4.9 requires you to evaluate and migrate your APIs as the latest version of Kubernetes has removed a significant number of APIs.

Before you can upgrade your Red Hat OpenShift Service on AWS clusters, you must update the required tools to the appropriate version.

Requirements for upgrading to OpenShift 4.9

You must meet the following requirements before upgrading a Red Hat OpenShift Service on AWS (ROSA) cluster that uses the AWS Security Token Service (STS) from version 4.8 to 4.9.

Prerequisites

You have installed the latest AWS CLI on your installation host.
You have installed 1.1.10 or later of the ROSA CLI on your installation host.
You have installed version 4.9 or later of the OpenShift CLI (oc) on your workstation(s) as needed.
You have the permissions required to update the AWS account-wide roles and policies.
You have access to the cluster as a user with the cluster-admin role.
You updated the AWS Identity and Access Management (IAM) account-wide roles and policies, including the Operator policies to version 4.9.

Administrator acknowledgment when upgrading to OpenShift 4.9

Red Hat OpenShift Service on AWS 4.9 uses Kubernetes 1.22, which removed a significant number of deprecated v1beta1 APIs.

Red Hat OpenShift Service on AWS 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from Red Hat OpenShift Service on AWS 4.8 to 4.9. This is to help prevent issues after upgrading to Red Hat OpenShift Service on AWS 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All Red Hat OpenShift Service on AWS 4.8 clusters require this administrator acknowledgment before they can be upgraded to Red Hat OpenShift Service on AWS 4.9.

Removed Kubernetes APIs

Red Hat OpenShift Service on AWS 4.9 uses Kubernetes 1.22, which removed the following deprecated v1beta1 APIs. You must migrate manifests and API clients to use the v1 API version. For more information about migrating removed APIs, see the Kubernetes documentation.

Table 1. `v1beta1` APIs removed from Kubernetes 1.22
Resource	API	Notable changes
APIService	apiregistration.k8s.io/v1beta1	No
CertificateSigningRequest	certificates.k8s.io/v1beta1	Yes
ClusterRole	rbac.authorization.k8s.io/v1beta1	No
ClusterRoleBinding	rbac.authorization.k8s.io/v1beta1	No
CSIDriver	storage.k8s.io/v1beta1	No
CSINode	storage.k8s.io/v1beta1	No
CustomResourceDefinition	apiextensions.k8s.io/v1beta1	Yes
Ingress	extensions/v1beta1	Yes
Ingress	networking.k8s.io/v1beta1	Yes
IngressClass	networking.k8s.io/v1beta1	No
Lease	coordination.k8s.io/v1beta1	No
LocalSubjectAccessReview	authorization.k8s.io/v1beta1	Yes
MutatingWebhookConfiguration	admissionregistration.k8s.io/v1beta1	Yes
PriorityClass	scheduling.k8s.io/v1beta1	No
Role	rbac.authorization.k8s.io/v1beta1	No
RoleBinding	rbac.authorization.k8s.io/v1beta1	No
SelfSubjectAccessReview	authorization.k8s.io/v1beta1	Yes
StorageClass	storage.k8s.io/v1beta1	No
SubjectAccessReview	authorization.k8s.io/v1beta1	Yes
TokenReview	authentication.k8s.io/v1beta1	No
ValidatingWebhookConfiguration	admissionregistration.k8s.io/v1beta1	Yes
VolumeAttachment	storage.k8s.io/v1beta1	No

Evaluating your cluster for removed APIs

There are several methods to help administrators identify where APIs that will be removed are in use. However, Red Hat OpenShift Service on AWS cannot identify all instances, especially workloads that are idle or external tools that are used. It is the responsibility of the administrator to properly evaluate all workloads and other integrations for instances of removed APIs.

Reviewing alerts to identify uses of removed APIs

The APIRemovedInNextReleaseInUse alert tells you that there are removed APIs in use on your cluster. If this alert is firing in your cluster, review the alert; take action to clear the alert by migrating manifests and API clients to use the new API version. You can use the APIRequestCount API to get more information about which APIs are in use and which workloads are using removed APIs.

Using APIRequestCount to identify uses of removed APIs

You can use the APIRequestCount API to track API requests and review if any of them are using one of the removed APIs.

Prerequisites

You must have access to the cluster as a user with the cluster-admin role.

Procedure

Run the following command and examine the REMOVEDINRELEASE column of the output to identify the removed APIs that are currently in use:

$ oc get apirequestcounts

Example output

NAME                                        REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
cloudcredentials.v1.operator.openshift.io                      32                      111
ingresses.v1.networking.k8s.io                                 28                      110
ingresses.v1beta1.extensions                1.22               16                      66
ingresses.v1beta1.networking.k8s.io         1.22               0                       1
installplans.v1alpha1.operators.coreos.com                     93                      167
...

You can safely ignore the following entries that appear in the results:

system:serviceaccount:kube-system:generic-garbage-collector appears in the results because it walks through all registered APIs searching for resources to remove.
system:kube-controller-manager appears in the results because it walks through all resources to count them while enforcing quotas.

You can also use -o jsonpath to filter the results:

$ oc get apirequestcounts -o jsonpath='{range .items[?(@.status.removedInRelease!="")]}{.status.removedInRelease}{"\t"}{.metadata.name}{"\n"}{end}'

Example output

1.22    certificatesigningrequests.v1beta1.certificates.k8s.io
1.22    ingresses.v1beta1.extensions
1.22    ingresses.v1beta1.networking.k8s.io

Using APIRequestCount to identify which workloads are using the removed APIs

You can examine the APIRequestCount resource for a given API version to help identify which workloads are using the API.

Prerequisites

You must have access to the cluster as a user with the cluster-admin role.

Procedure

Run the following command and examine the username and userAgent fields to help identify the workloads that are using the API:

$ oc get apirequestcounts <resource>.<version>.<group> -o yaml

For example:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io -o yaml

You can also use -o jsonpath to extract the username values from an APIRequestCount resource:

$ oc get apirequestcounts ingresses.v1beta1.networking.k8s.io -o jsonpath='{range ..username}{$}{"\n"}{end}' | sort | uniq

Example output

user1
user2
app:serviceaccount:delta

Migrating instances of removed APIs

For information on how to migrate removed Kubernetes APIs, see the Deprecated API Migration Guide in the Kubernetes documentation.