Preparing to update a cluster with manually maintained credentials - Preparing to update a cluster | Updating clusters

Update requirements for clusters with manually maintained credentials
Extracting and preparing credentials request resources
Configuring the Cloud Credential Operator utility for a cluster update
Updating cloud provider resources with the Cloud Credential Operator utility
Manually updating cloud provider resources
Indicating that the cluster is ready to upgrade

The Cloud Credential Operator (CCO) Upgradable status for a cluster with manually maintained credentials is False by default.

For minor releases, for example, from 4.12 to 4.13, this status prevents you from updating until you have addressed any updated permissions and annotated the CloudCredential resource to indicate that the permissions are updated as needed for the next version. This annotation changes the Upgradable status to True.
For z-stream releases, for example, from 4.13.0 to 4.13.1, no permissions are added or changed, so the update is not blocked.

Before updating a cluster with manually maintained credentials, you must accommodate any new or changed credentials in the release image for the version of OpenShift Container Platform you are updating to.

Update requirements for clusters with manually maintained credentials

Before you update a cluster that uses manually maintained credentials with the Cloud Credential Operator (CCO), you must update the cloud provider resources for the new release.

If the cloud credential management for your cluster was configured using the CCO utility (ccoctl), use the ccoctl utility to update the resources. Clusters that were configured to use manual mode without the ccoctl utility require manual updates for the resources.

After updating the cloud provider resources, you must update the upgradeable-to annotation for the cluster to indicate that it is ready to update.

The process to update the cloud provider resources and the upgradeable-to annotation can only be completed by using command line tools.

Cloud credential configuration options and update requirements by platform type

Some platforms only support using the CCO in one mode. For clusters that are installed on those platforms, the platform type determines the credentials update requirements.

For platforms that support using the CCO in multiple modes, you must determine which mode the cluster is configured to use and take the required actions for that configuration.

Decision tree showing the possible update paths for your cluster depending on the configured CCO credentials mode.

Figure 1. Credentials update requirements by platform type

Red Hat OpenStack Platform (RHOSP) and VMware vSphere

These platforms do not support using the CCO in manual mode. Clusters on these platforms handle changes in cloud provider resources automatically and do not require an update to the upgradeable-to annotation.

Administrators of clusters on these platforms should skip the manually maintained credentials section of the update process.

Alibaba Cloud, IBM Cloud, and Nutanix

Clusters installed on these platforms are configured using the ccoctl utility.

Administrators of clusters on these platforms must take the following actions:

Extract and prepare the CredentialsRequest custom resources (CRs) for the new release.
Configure the ccoctl utility for the new release and use it to update the cloud provider resources.
Indicate that the cluster is ready to update with the upgradeable-to annotation.

Microsoft Azure Stack Hub

These clusters use manual mode with long-term credentials and do not use the ccoctl utility.

Administrators of clusters on these platforms must take the following actions:

Extract and prepare the CredentialsRequest custom resources (CRs) for the new release.
Manually update the cloud provider resources for the new release.
Indicate that the cluster is ready to update with the upgradeable-to annotation.

Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP)

Clusters installed on these platforms support multiple CCO modes.

The required update process depends on the mode that the cluster is configured to use. If you are not sure what mode the CCO is configured to use on your cluster, you can use the web console or the CLI to determine this information.

Additional resources

Determining the Cloud Credential Operator mode by using the web console

You can determine what mode the Cloud Credential Operator (CCO) is configured to use by using the web console.

Only Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP) clusters support multiple CCO modes.

Prerequisites

You have access to an OpenShift Container Platform account with cluster administrator permissions.

Procedure

Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.
Navigate to Administration → Cluster Settings.
On the Cluster Settings page, select the Configuration tab.
Under Configuration resource, select CloudCredential.
On the CloudCredential details page, select the YAML tab.

In the YAML block, check the value of spec.credentialsMode. The following values are possible, though not all are supported on all platforms:

'': The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.
Mint: The CCO is operating in mint mode.
Passthrough: The CCO is operating in passthrough mode.
Manual: The CCO is operating in manual mode.

To determine the specific configuration of an AWS, GCP, or global Microsoft Azure cluster that has a spec.credentialsMode of '', Mint, or Manual, you must investigate further.

AWS and GCP clusters support using mint mode with the root secret deleted. If the cluster is specifically configured to use mint mode or uses mint mode by default, you must determine if the root secret is present on the cluster before updating.

An AWS, GCP, or global Microsoft Azure cluster that uses manual mode might be configured to create and manage cloud credentials from outside of the cluster with AWS STS, GCP Workload Identity, or Azure AD Workload Identity. You can determine whether your cluster uses this strategy by examining the cluster Authentication object.

AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, navigate to Workloads → Secrets and look for the root secret for your cloud provider.

Ensure that the Project dropdown is set to All Projects.

Platform Secret name

AWS

aws-creds

GCP

gcp-credentials
- If you see one of these values, your cluster is using mint or passthrough mode with the root secret present.
- If you do not see these values, your cluster is using the CCO in mint mode with the root secret removed.
AWS, GCP, or global Microsoft Azure clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, you must check the cluster Authentication object YAML values.
1. Navigate to Administration → Cluster Settings.
2. On the Cluster Settings page, select the Configuration tab.
3. Under Configuration resource, select Authentication.
4. On the Authentication details page, select the YAML tab.
5. In the YAML block, check the value of the .spec.serviceAccountIssuer parameter.
  A value that contains a URL that is associated with your cloud provider indicates that the CCO is using manual mode with short-term credentials for components. These clusters are configured using the ccoctl utility to create and manage cloud credentials from outside of the cluster.
  
  An empty value ('') indicates that the cluster is using the CCO in manual mode but was not configured using the ccoctl utility.

Platform	Secret name
AWS	`aws-creds`
GCP	`gcp-credentials`

Next steps

If you are updating a cluster that has the CCO operating in mint or passthrough mode and the root secret is present, you do not need to update any cloud provider resources and can continue to the next part of the update process.
If your cluster is using the CCO in mint mode with the root secret removed, you must reinstate the credential secret with the administrator-level credential before continuing to the next part of the update process.
If your cluster was configured using the CCO utility (ccoctl), you must take the following actions:
1. Extract and prepare the CredentialsRequest custom resources (CRs) for the new release.
2. Configure the ccoctl utility for the new release and use it to update the cloud provider resources.
3. Update the upgradeable-to annotation to indicate that the cluster is ready to update.
If your cluster is using the CCO in manual mode but was not configured using the ccoctl utility, you must take the following actions:
1. Extract and prepare the CredentialsRequest custom resources (CRs) for the new release.
2. Manually update the cloud provider resources for the new release.
3. Update the upgradeable-to annotation to indicate that the cluster is ready to update.

Additional resources

Extracting and preparing credentials request resources

Determining the Cloud Credential Operator mode by using the CLI

You can determine what mode the Cloud Credential Operator (CCO) is configured to use by using the CLI.

Only Amazon Web Services (AWS), global Microsoft Azure, and Google Cloud Platform (GCP) clusters support multiple CCO modes.

Prerequisites

You have access to an OpenShift Container Platform account with cluster administrator permissions.
You have installed the OpenShift CLI (oc).

Procedure

To determine the mode that the CCO is configured to use, enter the following command:

$ oc get cloudcredentials cluster \
  -o=jsonpath={.spec.credentialsMode}

The following output values are possible, though not all are supported on all platforms:

'': The CCO is operating in the default mode. In this configuration, the CCO operates in mint or passthrough mode, depending on the credentials provided during installation.
Mint: The CCO is operating in mint mode.
Passthrough: The CCO is operating in passthrough mode.
Manual: The CCO is operating in manual mode.

To determine the specific configuration of an AWS, GCP, or global Microsoft Azure cluster that has a spec.credentialsMode of '', Mint, or Manual, you must investigate further.

AWS or GCP clusters that use mint mode only: To determine whether the cluster is operating without the root secret, run the following command:
```
$ oc get secret <secret_name> \
  -n=kube-system
```
where <secret_name> is aws-creds for AWS or gcp-credentials for GCP.

If the root secret is present, the output of this command returns information about the secret. An error indicates that the root secret is not present on the cluster.
AWS, GCP, or global Microsoft Azure clusters that use manual mode only: To determine whether the cluster is configured to create and manage cloud credentials from outside of the cluster, run the following command:
```
$ oc get authentication cluster \
  -o jsonpath \
  --template='{ .spec.serviceAccountIssuer }'
```
This command displays the value of the .spec.serviceAccountIssuer parameter in the cluster Authentication object.
- An output of a URL that is associated with your cloud provider indicates that the CCO is using manual mode with short-term credentials for components. These clusters are configured using the ccoctl utility to create and manage cloud credentials from outside of the cluster.
- An empty output indicates that the cluster is using the CCO in manual mode but was not configured using the ccoctl utility.

Next steps

If you are updating a cluster that has the CCO operating in mint or passthrough mode and the root secret is present, you do not need to update any cloud provider resources and can continue to the next part of the update process.
If your cluster is using the CCO in mint mode with the root secret removed, you must reinstate the credential secret with the administrator-level credential before continuing to the next part of the update process.
If your cluster was configured using the CCO utility (ccoctl), you must take the following actions:
1. Extract and prepare the CredentialsRequest custom resources (CRs) for the new release.
2. Configure the ccoctl utility for the new release and use it to update the cloud provider resources.
3. Update the upgradeable-to annotation to indicate that the cluster is ready to update.
If your cluster is using the CCO in manual mode but was not configured using the ccoctl utility, you must take the following actions:
1. Extract and prepare the CredentialsRequest custom resources (CRs) for the new release.
2. Manually update the cloud provider resources for the new release.
3. Update the upgradeable-to annotation to indicate that the cluster is ready to update.

Additional resources

Extracting and preparing credentials request resources

Extracting and preparing credentials request resources

Before updating a cluster that uses the Cloud Credential Operator (CCO) in manual mode, you must extract and prepare the CredentialsRequest custom resources (CRs) for the new release.

Prerequisites

Install the OpenShift CLI (oc) that matches the version for your updated version.
Log in to the cluster as user with cluster-admin privileges.

Procedure

Obtain the pull spec for the update that you want to apply by running the following command:
```
$ oc adm upgrade
```
The output of this command includes pull specs for the available updates similar to the following:
Partial example output
```
...
Recommended updates:

VERSION IMAGE
4.15.0  quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032
...
```
Set a $RELEASE_IMAGE variable with the release image that you want to use by running the following command:
```
$ RELEASE_IMAGE=<update_pull_spec>
```
where <update_pull_spec> is the pull spec for the release image that you want to use. For example:
```
quay.io/openshift-release-dev/ocp-release@sha256:6a899c54dda6b844bb12a247e324a0f6cde367e880b73ba110c056df6d018032
```

Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image by running the following command:

$ oc adm release extract \
  --from=$RELEASE_IMAGE \
  --credentials-requests \
  --included \(1)
  --to=<path_to_directory_for_credentials_requests> (2)

1	The `--included` parameter includes only the manifests that your specific cluster configuration requires for the target release.
2	Specify the path to the directory where you want to store the `CredentialsRequest` objects. If the specified directory does not exist, this command creates it.

This command creates a YAML file for each CredentialsRequest object.

For each CredentialsRequest CR in the release image, ensure that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster. This field is where the generated secrets that hold the credentials configuration are stored.

Sample AWS CredentialsRequest object

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: cloud-credential-operator-iam-ro
  namespace: openshift-cloud-credential-operator
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - effect: Allow
      action:
      - iam:GetUser
      - iam:GetUserPolicy
      - iam:ListAccessKeys
      resource: "*"
  secretRef:
    name: cloud-credential-operator-iam-ro-creds
    namespace: openshift-cloud-credential-operator (1)

1	This field indicates the namespace which must exist to hold the generated secret.

The CredentialsRequest CRs for other platforms have a similar format with different platform-specific values.

For any CredentialsRequest CR for which the cluster does not already have a namespace with the name specified in spec.secretRef.namespace, create the namespace by running the following command:
```
$ oc create namespace <component_namespace>
```

Next steps

If the cloud credential management for your cluster was configured using the CCO utility (ccoctl), configure the ccoctl utility for a cluster update and use it to update your cloud provider resources.
If your cluster was not configured with the ccoctl utility, manually update your cloud provider resources.

Additional resources

Configuring the Cloud Credential Operator utility for a cluster update

To upgrade a cluster that uses the Cloud Credential Operator (CCO) in manual mode to create and manage cloud credentials from outside of the cluster, extract and prepare the CCO utility (ccoctl) binary.

The ccoctl utility is a Linux binary that must run in a Linux environment.

Prerequisites

You have access to an OpenShift Container Platform account with cluster administrator access.
You have installed the OpenShift CLI (oc).

Your cluster was configured using the ccoctl utility to create and manage cloud credentials from outside of the cluster.
You have extracted the CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image and ensured that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster.

Procedure

Obtain the CCO container image from the OpenShift Container Platform release image by running the following command:
```
$ CCO_IMAGE=$(oc adm release info --image-for='cloud-credential-operator' $RELEASE_IMAGE -a ~/.pull-secret)
```
Ensure that the architecture of the $RELEASE_IMAGE matches the architecture of the environment in which you will use the ccoctl tool.
Extract the ccoctl binary from the CCO container image within the OpenShift Container Platform release image by running the following command:
```
$ oc image extract $CCO_IMAGE --file="/usr/bin/ccoctl" -a ~/.pull-secret
```
Change the permissions to make ccoctl executable by running the following command:
```
$ chmod 775 ccoctl
```

Verification

To verify that ccoctl is ready to use, display the help file by running the following command:

$ ccoctl --help

Output of ccoctl --help

OpenShift credentials provisioning tool

Usage:
  ccoctl [command]

Available Commands:
  alibabacloud Manage credentials objects for alibaba cloud
  aws          Manage credentials objects for AWS cloud
  azure        Manage credentials objects for Azure
  gcp          Manage credentials objects for Google cloud
  help         Help about any command
  ibmcloud     Manage credentials objects for IBM Cloud
  nutanix      Manage credentials objects for Nutanix

Flags:
  -h, --help   help for ccoctl

Use "ccoctl [command] --help" for more information about a command.

Updating cloud provider resources with the Cloud Credential Operator utility

The process for upgrading an OpenShift Container Platform cluster that was configured using the CCO utility (ccoctl) is similar to creating the cloud provider resources during installation.

On AWS clusters, some ccoctl commands make AWS API calls to create or modify AWS resources. You can use the --dry-run flag to avoid making API calls. Using this flag creates JSON files on the local file system instead. You can review and modify the JSON files and then apply them with the AWS CLI tool using the --cli-input-json parameters.

Prerequisites

You have extracted the CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image and ensured that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster.
You have extracted and configured the ccoctl binary from the release image.

Procedure

Use the ccoctl tool to process all CredentialsRequest objects by running the command for your cloud provider. The following commands process CredentialsRequest objects:

Alibaba Cloud

$ ccoctl alibabacloud create-ram-users \
  --name <name> \(1)
  --region=<alibaba_region> \(2)
  --credentials-requests-dir=<path_to_credentials_requests_directory> \(3)
  --output-dir=<path_to_ccoctl_output_dir> (4)

1	Specify the name used to tag any cloud resources that are created for tracking.
2	Specify the Alibaba Cloud region in which cloud resources will be created.
3	Specify the directory containing the files for the component `CredentialsRequest` objects.
4	Optional: Specify the directory in which you want the `ccoctl` utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

A RAM user can have up to two AccessKeys at the same time. If you run ccoctl alibabacloud create-ram-users more than twice, the previously generated manifests secret becomes stale and you must reapply the newly generated secrets.

Amazon Web Services (AWS)

$ ccoctl aws create-all \(1)
  --name=<name> \(2)
  --region=<aws_region> \(3)
  --credentials-requests-dir=<path_to_credentials_requests_directory> \(4)
  --output-dir=<path_to_ccoctl_output_dir> \(5)
  --create-private-s3-bucket (6)

1	To create the AWS resources individually, use the "Creating AWS resources individually" procedure in the "Installing a cluster on AWS with customizations" content. This option might be useful if you need to review the JSON files that the `ccoctl` tool creates before modifying AWS resources, or if the process the `ccoctl` tool uses to create AWS resources automatically does not meet the requirements of your organization.
2	Specify the name used to tag any cloud resources that are created for tracking.
3	Specify the AWS region in which cloud resources will be created.
4	Specify the directory containing the files for the component `CredentialsRequest` objects.
5	Optional: Specify the directory in which you want the `ccoctl` utility to create objects. By default, the utility creates objects in the directory in which the commands are run.
6	Optional: By default, the `ccoctl` utility stores the OpenID Connect (OIDC) configuration files in a public S3 bucket and uses the S3 URL as the public OIDC endpoint. To store the OIDC configuration in a private S3 bucket that is accessed by the IAM identity provider through a public CloudFront distribution URL instead, use the `--create-private-s3-bucket` parameter.

Google Cloud Platform (GCP)

$ ccoctl gcp create-all \
  --name=<name> \(1)
  --region=<gcp_region> \(2)
  --project=<gcp_project_id> \(3)
  --credentials-requests-dir=<path_to_credentials_requests_directory> \(4)
  --output-dir=<path_to_ccoctl_output_dir> (5)

1	Specify the user-defined name for all created GCP resources used for tracking.
2	Specify the GCP region in which cloud resources will be created.
3	Specify the GCP project ID in which cloud resources will be created.
4	Specify the directory containing the files of `CredentialsRequest` manifests to create GCP service accounts.
5	Optional: Specify the directory in which you want the `ccoctl` utility to create objects. By default, the utility creates objects in the directory in which the commands are run.

IBM Cloud

$ ccoctl ibmcloud create-service-id \
  --credentials-requests-dir=<path_to_credential_requests_directory> \(1)
  --name=<cluster_name> \(2)
  --output-dir=<installation_directory> \(3)
  --resource-group-name=<resource_group_name> (4)

1	Specify the directory containing the files for the component `CredentialsRequest` objects.
2	Specify the name of the OpenShift Container Platform cluster.
3	Optional: Specify the directory in which you want the `ccoctl` utility to create objects. By default, the utility creates objects in the directory in which the commands are run.
4	Optional: Specify the name of the resource group used for scoping the access policies.

Nutanix

$ ccoctl nutanix create-shared-secrets \
  --credentials-requests-dir=<path_to_credentials_requests_directory> \(1)
  --output-dir=<ccoctl_output_dir> \(2)
  --credentials-source-filepath=<path_to_credentials_file> (3)

1	Specify the path to the directory that contains the files for the component `CredentialsRequests` objects.
2	Optional: Specify the directory in which you want the `ccoctl` utility to create objects. By default, the utility creates objects in the directory in which the commands are run.
3	Optional: Specify the directory that contains the credentials data YAML file. By default, `ccoctl` expects this file to be in `<home_directory>/.nutanix/credentials`.

For each CredentialsRequest object, ccoctl creates the required provider resources and a permissions policy as defined in each CredentialsRequest object from the OpenShift Container Platform release image.

Apply the secrets to your cluster by running the following command:

$ ls <path_to_ccoctl_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}

Verification

You can verify that the required provider resources and permissions policies are created by querying the cloud provider. For more information, refer to your cloud provider documentation on listing roles or service accounts.

Next steps

Update the upgradeable-to annotation to indicate that the cluster is ready to upgrade.

Additional resources

Indicating that the cluster is ready to upgrade

Manually updating cloud provider resources

Before upgrading a cluster with manually maintained credentials, you must create secrets for any new credentials for the release image that you are upgrading to. You must also review the required permissions for existing credentials and accommodate any new permissions requirements in the new release for those components.

Prerequisites

You have extracted the CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image and ensured that a namespace that matches the text in the spec.secretRef.namespace field exists in the cluster.

Procedure

Create YAML files with secrets for any CredentialsRequest custom resources that the new release image adds. The secrets must be stored using the namespace and secret name defined in the spec.secretRef for each CredentialsRequest object.

Sample AWS YAML files

Sample AWS CredentialsRequest object with secrets

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: <component_credentials_request>
  namespace: openshift-cloud-credential-operator
  ...
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - effect: Allow
      action:
      - s3:CreateBucket
      - s3:DeleteBucket
      resource: "*"
      ...
  secretRef:
    name: <component_secret>
    namespace: <component_namespace>
  ...

Sample AWS Secret object

apiVersion: v1
kind: Secret
metadata:
  name: <component_secret>
  namespace: <component_namespace>
data:
  aws_access_key_id: <base64_encoded_aws_access_key_id>
  aws_secret_access_key: <base64_encoded_aws_secret_access_key>

Sample Azure YAML files

Global Azure and Azure Stack Hub use the same CredentialsRequest object and secret formats.

Sample Azure CredentialsRequest object with secrets

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: <component_credentials_request>
  namespace: openshift-cloud-credential-operator
  ...
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AzureProviderSpec
    roleBindings:
    - role: Contributor
      ...
  secretRef:
    name: <component_secret>
    namespace: <component_namespace>
  ...

Sample Azure Secret object

apiVersion: v1
kind: Secret
metadata:
  name: <component_secret>
  namespace: <component_namespace>
data:
  azure_subscription_id: <base64_encoded_azure_subscription_id>
  azure_client_id: <base64_encoded_azure_client_id>
  azure_client_secret: <base64_encoded_azure_client_secret>
  azure_tenant_id: <base64_encoded_azure_tenant_id>
  azure_resource_prefix: <base64_encoded_azure_resource_prefix>
  azure_resourcegroup: <base64_encoded_azure_resourcegroup>
  azure_region: <base64_encoded_azure_region>

Sample GCP YAML files

Sample GCP CredentialsRequest object with secrets

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: <component_credentials_request>
  namespace: openshift-cloud-credential-operator
  ...
spec:
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: GCPProviderSpec
      predefinedRoles:
      - roles/iam.securityReviewer
      - roles/iam.roleViewer
      skipServiceCheck: true
      ...
  secretRef:
    name: <component_secret>
    namespace: <component_namespace>
  ...

Sample GCP Secret object

apiVersion: v1
kind: Secret
metadata:
  name: <component_secret>
  namespace: <component_namespace>
data:
  service_account.json: <base64_encoded_gcp_service_account_file>

If the CredentialsRequest custom resources for any existing credentials that are stored in secrets have changed permissions requirements, update the permissions as required.

Next steps

Update the upgradeable-to annotation to indicate that the cluster is ready to upgrade.

Additional resources

Indicating that the cluster is ready to upgrade

The Cloud Credential Operator (CCO) Upgradable status for a cluster with manually maintained credentials is False by default.

Prerequisites

For the release image that you are upgrading to, you have processed any new credentials manually or by using the Cloud Credential Operator utility (ccoctl).
You have installed the OpenShift CLI (oc).

Procedure

Log in to oc on the cluster as a user with the cluster-admin role.
Edit the CloudCredential resource to add an upgradeable-to annotation within the metadata field by running the following command:
```
$ oc edit cloudcredential cluster
```
Text to add
```
...
  metadata:
    annotations:
      cloudcredential.openshift.io/upgradeable-to: <version_number>
...
```
Where <version_number> is the version that you are upgrading to, in the format x.y.z. For example, use 4.12.2 for OpenShift Container Platform 4.12.2.

It may take several minutes after adding the annotation for the upgradeable status to change.

Verification

In the Administrator perspective of the web console, navigate to Administration → Cluster Settings.
To view the CCO status details, click cloud-credential in the Cluster Operators list.
- If the Upgradeable status in the Conditions section is False, verify that the upgradeable-to annotation is free of typographical errors.
When the Upgradeable status in the Conditions section is True, begin the OpenShift Container Platform upgrade.