OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Creating ROSA with HCP clusters with external authentication

edit

You can create Red Hat OpenShift Service on AWS (ROSA) with hosted control planes (HCP) clusters that use an external OpenID Connect (OIDC) identity provider to issue tokens for authentication, replacing the built-in OpenShift OAuth server. While the built-in OpenShift OAuth server supports integration with a variety of identity providers, including external OIDC identity providers, it is limited to the capabilities of the OAuth server itself. You can directly integrate external OIDC identity providers with ROSA with HCP clusters in order to facilitate machine-to-machine workflows, such as CLI, and provide additional capabilities which are not available when using the built-in OpenShift OAuth server.

Since it is not possible to upgrade or convert existing ROSA clusters to a hosted control planes architecture, you must create a new cluster to use ROSA with HCP functionality. You also cannot convert a cluster that was created to use external authentication providers to use the internal OAuth2 server. You must also create a new cluster.

Sharing VPCs across multiple AWS accounts is not currently supported for ROSA with HCP. Do not install a ROSA with HCP cluster into subnets shared from another AWS account. See "Are multiple ROSA clusters in a single VPC supported?" for more information.

ROSA with HCP clusters only support Security Token Service (STS) authentication.
Further reading
edit

ROSA with HCP Prerequisites

To create a ROSA with HCP cluster, you must have completed the following steps:

edit

Creating a ROSA with HCP cluster that uses external authentication providers

Use the --external-auth-providers-enabled flag in the ROSA CLI to create a cluster that uses an external authentication service.

When creating a ROSA with HCP cluster, the default machine Classless Inter-Domain Routing (CIDR) is 10.0.0.0/16. If this does not correspond to the CIDR range for your VPC subnets, add --machine-cidr <address_block> to the following commands.
Procedure

  • If you used the OIDC_ID, SUBNET_IDS, and OPERATOR_ROLES_PREFIX variables to prepare your environment, you can continue to use those variables when creating your cluster. For example, run the following command:

    $ rosa create cluster --hosted-cp --subnet-ids=$SUBNET_IDS \
   --oidc-config-id=$OIDC_ID --cluster-name=<cluster_name> \
   --operator-roles-prefix=$OPERATOR_ROLES_PREFIX \
   --external-auth-providers-enabled

  • If you did not set environmental variables, run the following command:

    $ rosa create cluster --cluster-name=<cluster_name> --sts --mode=auto \
    --hosted-cp --operator-roles-prefix <operator-role-prefix> \
    --oidc-config-id <ID-of-OIDC-configuration> \
    --external-auth-providers-enabled \
    --subnet-ids=<public-subnet-id>,<private-subnet-id>
Verification

  • Verify that your external authentication is enabled in the cluster details by running the following command:

    $ rosa describe cluster --cluster=<cluster_name>
    Example output
    Name:                       rosa-ext-test
Display Name:               rosa-ext-test
ID:                         <cluster_id>
External ID:                <cluster_ext_id>
Control Plane:              ROSA Service Hosted
OpenShift Version:          4.17.0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <AWS_id>
AWS Billing Account:        <AWS_id>
API URL:                    <ocm_api>
Console URL:
Region:                     us-east-1
Availability:
 - Control Plane:           MultiAZ
 - Data Plane:              SingleAZ

Nodes:
 - Compute (desired):       2
 - Compute (current):       0
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             /23
 - Subnets:                 <subnet_ids>
EC2 Metadata Http Tokens:   optional
Role (STS) ARN:             arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Installer-Role
Support Role ARN:           arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Support-Role
Instance IAM Roles:
 - Worker:                  arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Worker-Role
Operator IAM Roles:
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-cloud-network-config-controller-clo
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-capa-controller-manager
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-control-plane-operator
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-kms-provider
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-kube-controller-manager
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-image-registry-installer-cloud-cred
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-ingress-operator-cloud-credentials
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-cluster-csi-drivers-ebs-cloud-crede
Managed Policies:           Yes
State:                      ready
Private:                    No
Created:                    Mar 29 2024 14:25:52 UTC
User Workload Monitoring:   Enabled
Details Page:               https://<url>
OIDC Endpoint URL:          https://<endpoint> (Managed)
Audit Log Forwarding:       Disabled
External Authentication:    Enabled (1)
    1 The External Authentication flag is enabled, and you can now create an external authentication provider.
edit

Creating an external authentication provider

After you have created a ROSA with HCP cluster with the enabled option for external authentication providers, you must create a provider using the ROSA CLI.

Similar to the rosa create|delete|list idp[s] command in the ROSA CLI, you cannot edit an existing identity provider that you created using rosa create external-auth-provider. Instead, you must delete the external authentication provider and create a new one.

The following table shows the possible CLI flags you can use when creating your external authentication provider:

CLI Flag Description

--cluster

The name or the ID of your cluster.

--name

A name that is used to refer to the external authentication provider.

--console-client-secret

This string is the client secret that is used to associate your account with the application. If you do not include the client secret, this command uses a public OIDC OAuthClient.

--issuer-audiences

This is a comma-separated list of token audiences.

--issuer-url

The URL of the token issuer.

--claim-mapping-username-claim

The name of the claim that should be used to construct user names for the cluster identity.

--claim-mapping-groups-claim

The name of the claim that should be used to construct group names for the cluster identity.
Procedure

  • To use the interactive command interface, run the following commands:

    $ rosa create external-auth-provider -c <cluster_name>
    Example output
    I: Enabling interactive mode
? Name: (1)
? Issuer audiences: (2)
? The serving url of the token issuer: (3)
? CA file path (optional): (4)
? Claim mapping username: (5)
? Claim mapping groups: (6)
? Claim validation rule (optional): (7)
? Console client id (optional): (8)
    1 The name of your external authentication provider. This name should be a lower-case with numbers and dashes.
    2 The audience IDs that this authentication provider issues tokens for.
    3 The issuer’s URL that serves the token.
    4 Optional: The certificate file to use when making requests.
    5 The name of the claim that is used to construct the user names for cluster identity, such as using email.
    6 The method with which to transform the ID token into a cluster identity, such as using groups.
    7 Optional: The rules that help validate token claims which authenticate your users. This field should be formatted as :<required_value>.
    8 Optional: The application or client ID that your app registration uses for the console.

  • You can include the required IDs to create your external authentication provider with the following command:

    rosa create external-auth-provider --cluster=<cluster_id> \
    --name=<provider_name> --issuer-url=<issuing_url> \
    --issuer-audiences=<audience_id> \
    --claim-mapping-username-claim=email \
    --claim-mapping-groups-claim=groups \
    --console-client-id=<client_id_for_app_registration> \
    --console-client-secret=<client_secret>
    Example output
    I: Successfully created an external authentication provider for cluster '<cluster_id>'
Verification

  • To verify your external authentication provider, run one of the following options:

    • List the external authentication configuration on a specified cluster with the following command:

      $ rosa list external-auth-provider -c <cluster_name>
      Example output

      The following example shows a configured Microsoft Entra ID external authentication provider:

      NAME        ISSUER URL
m-entra-id  https://login.microsoftonline.com/<group_id>/v2.0

    • Display the external authentication configuration on a specified cluster by using the following command:

      $ rosa describe external-auth-provider \
    -c <cluster_name> --name <name_of_external_authentication>
      Example output
      ID:                          ms-entra-id
Cluster ID:                  <cluster_id>
Issuer audiences:
                             - <audience_id>
Issuer Url:                  https://login.microsoftonline.com/<group_id>/v2.0
Claim mappings group:        groups
Claim mappings username:     email
Additional resources
edit

Creating a break glass credential for a ROSA with HCP cluster

As a ROSA with HCP cluster owner, you can use the break glass credential to create temporary administrative client credentials to access your clusters that are configured with custom OpenID Connect (OIDC) token issuers. Creating a break glass credential generates a new cluster-admin kubeconfig file. The kubeconfig file contains information about the cluster that the CLI uses to connect a client to the correct cluster and API server. You can use the newly generated kubeconfig file to allow access to the ROSA with HCP cluster.

Prerequisites

  • You have created a ROSA with HCP cluster with external authentication enabled. For more information, see Creating a ROSA with HCP with HCP cluster that uses external authentication providers.

  • You have created an external authentication provider. For more information, see Creating an external authentication provider.

  • You have an account with cluster admin permissions.

Procedure

  1. Create a break glass credential by using one of the following commands:

    • To create a break glass credential by using the interactive command interface to interactively specify custom settings, run the following command:

      $ rosa create break-glass-credential -c <cluster_name> -i (1)
      1 Replace <cluster_name> with the name of your cluster.

      This command starts an interactive CLI process:

      Example output
      I: Enabling interactive mode
? Username (optional): (1)
? Expiration duration (optional): (2)
I: Successfully created a break glass credential for cluster 'ac-hcp-test'.
      1 If left blank, the value in the username will have a randomly generated username value.
      2 The minimum validity of the break glass credential is 10 minutes, and the maximum validity is 24 hours. If left blank, the expiration duration value defaults to 24 hours.

    • To create a break glass credential for cluster called mycluster with specified values:

      $ rosa create break-glass-credential -c mycluster --username test-username --expiration 1h

  2. List the break glass credential IDs, status, and associated users that are available for a cluster called mycluster by running the following command:

    $ rosa list break-glass-credential -c mycluster
    Example output
    ID                                USERNAME    STATUS
2a7jli9n4phe6c02ul7ti91djtv2o51d  test-user   issued

    You can also view the credentials in a JSON output by adding the -o json argument to the command.

  3. To view the status of a break glass credential, run the following command, replacing <break_glass_credential_id> with the break glass credential ID:

    $ rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name>
    Example output
    ID:                                    2a7jli9n4phe6c02ul7ti91djtv2o51d
Username:                              test-user
Expire at:                             Dec 28 2026 10:23:05 EDT
Status:                                issued

    The following is a list of possible Status field values:

    • issued The break glass credential has been issued and is ready to use.

    • expired The break glass credential has expired and can no longer be used.

    • failed The break glass credential has failed to create. In this case, you receive a service log detailing the failure. For more information about service logs, see Accessing the service logs for Red Hat OpenShift Service on AWS clusters. For steps to contact Red Hat Support for assistance, see Getting support.

    • awaiting_revocation The break glass credential is currently being revoked, meaning it cannot be used.

    • revoked The break glass credential has been revoked and can no longer be used.

  4. To retrieve the kubeconfig, run the following commands:

    • Create a kubeconfigs directory:

      $ mkdir ~/kubeconfigs

    • Export the newly generated kubeconfig file, replacing <cluster_name> with the name of your cluster:

      $ export CLUSTER_NAME=<cluster_name> && export KUBECONFIG=~/kubeconfigs/break-glass-${CLUSTER_NAME}.kubeconfig

    • View the kubeconfig:

      $ rosa describe break-glass-credential <break_glass_credential_id> -c mycluster --kubeconfig
      Example output
      apiVersion: v1
clusters:
- cluster:
    server: <server_url>
  name: cluster
contexts:
- context:
    cluster: cluster
    namespace: default
    user: test-username
  name: admin
current-context: admin
kind: Config
preferences: {}
users:
- name: test-user
  user:
    client-certificate-data: <client-certificate-data> (1)
    client-key-data: <client-key-data> (2)
      1 The client-certificate contains a certificate for the user signed by the Kubernetes certificate authorities (CA).
      2 The client-key contains the key that signed the client certificate.

  5. Optional: To save the kubeconfig, run the following command :

    $ rosa describe break-glass-credential <break_glass_credential_id> -c mycluster --kubeconfig > $KUBECONFIG
Additional resources
edit

Accessing a ROSA with HCP cluster by using a break glass credential

Use the new kubeconfig from the break glass credential to gain temporary admin access to a ROSA with HCP cluster.

Prerequisites

  • You have access to a ROSA with HCP cluster with external authentication enabled. For more information, see Creating a ROSA with HCP cluster that uses external authentication providers.

  • You have installed the oc and the kubectl CLIs.

  • You have configured the new kubeconfig. For more information, see Creating a break glass credential for a ROSA with HCP cluster.

Procedure

  1. Access the details for the cluster:

    $ rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name>  --kubeconfig > $KUBECONFIG

  2. List the nodes from the cluster:

    $ oc get nodes
    Example output
    NAME                        STATUS   ROLES   AGE   VERSION
ip-10-0-0-27.ec2.internal   Ready    worker  8m    v1.28.7+f1b5f6c
ip-10-0-0-67.ec2.internal   Ready    worker  9m    v1.28.7+f1b5f6c

  3. Verify you have the correct credentials:

    $ kubectl auth whoami
    Example output
    ATTRIBUTE    VALUE
Username     system:customer-break-glass:test-user
Groups       [system:masters system:authenticated]

  4. Apply the ClusterRoleBinding for the groups defined in the external OIDC provider. The ClusterRoleBinding maps the rosa-hcp-admins group that is created in Microsoft Entra ID to a group in the ROSA with HCP cluster.

    $ oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: rosa-hcp-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: f715c264-ab90-45d5-8a29-2e91a609a895
EOF
    Example output
    clusterrolebinding.rbac.authorization.k8s.io/rosa-hcp-admins created

    After the ClusterRoleBinding has been applied, the ROSA with HCP cluster is configured, and the rosa CLI and the Red Hat Hybrid Cloud Console are authenticated through the external OpenID Connect (OIDC) provider. You can now start assigning roles and deploying applications on the cluster.
Additional resources
edit

Revoking a break glass credential for a ROSA with HCP cluster

You can revoke access to any break glass credentials that you have provisioned at any time by using the revoke break-glass-credentials command.

Prerequisites

  • You have created a break glass credential.

  • You are the cluster owner.

Procedure

  • Revoke the break glass credentials for a ROSA with HCP cluster by running the following command.

    Running this command will revoke access for all break glass credentials related to the cluster.

    		 
    $ rosa revoke break-glass-credentials -c <cluster_name> (1)
    1 Replace <cluster_name> with the name of your cluster.
    Example output
    ? Are you sure you want to revoke all the break glass credentials on cluster 'my-cluster'?: Yes
I: Successfully requested revocation for all break glass credentials from cluster 'my-cluster'
Verification

  • The revocation process can take several minutes. You can verify that the break glass credentials for your clusters have been revoked by running one of the following commands:

    • List all break glass credentials and check the status of each:

      $ rosa list break-glass-credential -c <cluster_name>
      Example output
      ID                                USERNAME    STATUS
2330dbs0n8m3chkkr25gkkcd8pnj3lk2  test-user   awaiting_revocation

    • You can also verify the status by checking the individual credential:

      $ rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name>
      Example output
      ID:                                    2330dbs0n8m3chkkr25gkkcd8pnj3lk2
Username:                              test-user
Expire at:                             Dec 28 2026 10:23:05 EDT
Status:                                issued
Revoked at:                            Dec 27 2026 15:30:33 EDT
edit

Deleting an external authentication provider

Delete external authentication providers by using the ROSA CLI.

Procedure

  1. Display your external authentication provider on your cluster by running the following command:

    $ rosa list external-auth-provider -c <cluster_name>
    Example output
    NAME        ISSUER URL
entra-test  https://login.microsoftonline.com/<group_id>/v2.0

  2. Delete the external authentication provider by running the following command:

    $ rosa delete external-auth-provider <name_of_provider> -c <cluster_name>
    Example output
    ? Are you sure you want to delete external authentication provider entra-test on cluster rosa-ext-test? Yes
I: Successfully deleted external authentication provider 'entra-test' from cluster 'rosa-ext-test'
Verification

  1. Query for any external authentication providers on your cluster by running the following command:

    $ rosa list external-auth-provider -c <cluster_name>
    Example output
    E: there are no external authentication providers for this cluster

Additional resources