×

Use the Red Hat OpenShift Service on AWS CLI (rosa) to create an OpenShift cluster with the AWS Security Token Service (STS) using customizations.

Support considerations for ROSA clusters with STS

The supported way of creating a Red Hat OpenShift Service on AWS cluster that uses the AWS Security Token Service (STS) is by using the steps described in this product documentation.

You can use manual mode with the Red Hat OpenShift Service on AWS CLI (rosa) to generate the AWS Identity and Access Management (IAM) policy files and aws commands that are required to install the STS resources.

The files and aws commands are generated for review purposes only and must not be modified in any way. Red Hat cannot provide support for ROSA clusters that have been deployed by using modified versions of the policy files or aws commands.

Creating a cluster with STS using customizations

When you create a Red Hat OpenShift Service on AWS (ROSA) cluster that uses the AWS Security Token Service (STS), you can customize your installation interactively. When you run rosa create cluster --interactive at cluster creation time, you are presented with a series of interactive prompts that enable you to customize your deployment. For more information, see Interactive cluster creation mode reference.

There are two rosa CLI modes for deploying a cluster with STS:

  • manual mode. With this mode, the rosa CLI generates the aws commands needed to create the required AWS Identity and Access Management (IAM) roles and policies, and an OpenID Connect (OIDC) provider. The corresponding policy JSON files are also saved to the current directory. This enables you to review the details before running the aws commands manually.

  • auto mode. You can use this mode to immediately create the required AWS Identity and Access Management (IAM) resources using the current AWS account.

Only public and AWS PrivateLink clusters are supported with STS. Regular private clusters (non-PrivateLink) are not available for use with STS.

AWS Shared VPCs are not currently supported for ROSA installations.

Prerequisites
  • You have completed the AWS prerequisites for ROSA with STS.

  • You have available AWS service quotas.

  • You have enabled the ROSA service in the AWS Console.

  • You have installed and configured the latest AWS, ROSA, and oc CLIs on your installation host.

  • If you are using a customer-managed AWS Key Management Service (KMS) key for encryption, you have created a symmetric KMS key and you have the key ID and Amazon Resource Name (ARN). For more information about creating AWS KMS keys, see the AWS documenation.

Procedure
  1. Create the required account-wide roles and policies, including the Operator policies:

    1. Generate the IAM policy JSON files in the current working directory and output the aws CLI commands for review:

      $ rosa create account-roles --mode manual (1)
      1 manual mode generates the aws CLI commands and JSON files needed to create the account-wide roles and policies. After review, you must run the commands manually to create the resources.

      You can optionally specify an OpenShift minor release, for example 4.8, by using the --version option. The latest stable version is assumed if the option is not included. The account-wide roles and policies are specific to an OpenShift minor release version and are backward compatible.

    2. After review, run the aws commands manually to create the roles and policies. Alternatively, you can run the preceding command using --mode auto to run the aws commands immediately.

  2. (Optional) If you are using your own AWS KMS key to encrypt the control plane data volumes and the persistent volumes (PVs) for your applications, add the ARN for the account-wide installer role to your KMS key policy.

    1. Save the key policy for your KMS key to a file on your local machine. The following example saves the output to kms-key-policy.json in the current working directory:

      $ aws kms get-key-policy --key-id <key_id_or_arn> --policy-name default --output text > kms-key-policy.json (1)
      1 Replace <key_id_or_arn> with the ID or ARN of your KMS key.
    2. Add the ARN for the account-wide installer role that you created in the preceding step to the Statement.Principal.AWS section in the file. In the following example, the ARN for the default ManagedOpenShift-Installer-Role role is added:

      {
        "Version": "2012-10-17",
        "Id": "key-default-1",
        "Statement": [
          {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
              "AWS": [
                "arn:aws:iam::<aws_account_id>:root",
                "arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Installer-Role" (1)
              ]
            },
            "Action": "kms:*",
            "Resource": "*"
          }
        ]
      }
      1 You must specify the ARN for the account-wide role that will be used when you create the ROSA cluster. The ARNs listed in the section must be comma-separated.
    3. Apply the changes to your KMS key policy:

      $ aws kms put-key-policy --key-id <key_id_or_arn> \ (1)
          --policy file://kms-key-policy.json \ (2)
          --policy-name default
      1 Replace <key_id_or_arn> with the ID or ARN of your KMS key.
      2 You must include the file:// prefix when referencing a key policy in a local file.

      You can reference the ARN of your KMS key when you create the cluster in the next step.

  3. Create a cluster with STS using custom installation options. You can use the --interactive mode to interactively specify custom settings:

    $ rosa create cluster --interactive --sts
    Example output
    I: Interactive mode enabled.
    Any optional fields can be left empty and a default will be selected.
    ? Cluster name: <cluster_name>
    ? OpenShift version: 4.8.9 (1)
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Installer-Role for the Installer role (2)
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-ControlPlane-Role for the ControlPlane role
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Worker-Role for the Worker role
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Support-Role for the Support role
    ? External ID (optional):
    ? Operator roles prefix: <cluster_name>-<random_string>
    ? Multiple availability zones (optional): No (3)
    ? AWS region: us-east-1
    ? PrivateLink cluster (optional): No
    ? Install into an existing VPC (optional): No
    ? Enable Customer Managed key (optional): No (4)
    ? Compute nodes instance type (optional):
    ? Enable autoscaling (optional): No
    ? Compute nodes: 2
    ? Machine CIDR: 10.0.0.0/16
    ? Service CIDR: 172.30.0.0/16
    ? Pod CIDR: 10.128.0.0/14
    ? Host prefix: 23
    ? Disable Workload monitoring (optional): No
    I: Creating cluster '<cluster_name>'
    I: To create this cluster again in the future, you can run:
       rosa create cluster --cluster-name <cluster_name> --role-arn arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Installer-Role --support-role-arn arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Support-Role --master-iam-role arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-ControlPlane-Role --worker-iam-role arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Worker-Role --operator-roles-prefix <cluster_name>-<random_string> --region us-east-1 --version 4.8.9 --compute-nodes 2 --machine-cidr 10.0.0.0/16 --service-cidr 172.30.0.0/16 --pod-cidr 10.128.0.0/14 --host-prefix 23 (5)
    I: To view a list of clusters and their status, run 'rosa list clusters'
    I: Cluster '<cluster_name>' has been created.
    I: Once the cluster is installed you will need to add an Identity Provider before you can login into the cluster. See 'rosa create idp --help' for more information.
    I: To determine when your cluster is Ready, run 'rosa describe cluster -c <cluster_name>'.
    I: To watch your cluster installation logs, run 'rosa logs install -c <cluster_name> --watch'.
    1 When creating the cluster, the listed OpenShift version options include the major, minor, and patch versions, for example 4.8.9.
    2 If more than one matching set of account-wide roles are available in your account for a cluster version, an interactive list of options is provided.
    3 Multiple availability zones are recommended for production workloads. The default is a single availability zone.
    4 Enable this option if you are using your own AWS KMS key to encrypt the control plane data volumes and the PVs for your applications. Specify the ARN for the KMS key that you added the account-wide role ARN to in the preceding step.
    5 The output includes a custom command that you can run to create a cluster with the same configuration in the future.

    As an alternative to using the --interactive mode, you can specify the customization options directly when you run rosa create cluster. Run rosa create cluster --help to view a list of available CLI options.

    You must complete the following steps to create the Operator IAM roles and the OpenID Connect (OIDC) provider to move the state of the cluster to ready.

  4. Create the cluster-specific Operator IAM roles:

    1. Generate the Operator IAM policy JSON files in the current working directory and output the aws CLI commands for review:

      $ rosa create operator-roles --mode manual --cluster <cluster_name|cluster_id> (1)
      1 manual mode generates the aws CLI commands and JSON files needed to create the Operator roles. After review, you must run the commands manually to create the resources.
    2. After review, run the aws commands manually to create the Operator IAM roles and attach the managed Operator policies to them. Alternatively, you can run the preceding command again using --mode auto to run the aws commands immediately.

  5. Create the OpenID Connect (OIDC) provider that the cluster Operators use to authenticate:

    $ rosa create oidc-provider --mode auto --cluster <cluster_name|cluster_id> (1)
    1 auto mode immediately runs the aws CLI command that creates the OIDC provider.
  6. Check the status of your cluster:

    $ rosa describe cluster --cluster <cluster_name|cluster_id>
    Example output
    Name:                       <cluster_name>
    ID:                         <cluster_id>
    External ID:                <external_id>
    OpenShift Version:          <version>
    Channel Group:              stable
    DNS:                        <cluster_name>.xxxx.p1.openshiftapps.com
    AWS Account:                <aws_account_id>
    API URL:                    https://api.<cluster_name>.xxxx.p1.openshiftapps.com:6443
    Console URL:                https://console-openshift-console.apps.<cluster_name>.xxxx.p1.openshiftapps.com
    Region:                     <aws_region>
    Multi-AZ:                   false
    Nodes:
     - Master:                  3
     - Infra:                   2
     - Compute:                 2
    Network:
     - Service CIDR:            172.30.0.0/16
     - Machine CIDR:            10.0.0.0/16
     - Pod CIDR:                10.128.0.0/14
     - Host Prefix:             /23
    STS Role ARN:               arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Installer-Role
    Support Role ARN:           arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Support-Role
    Instance IAM Roles:
     - Master:                  arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-ControlPlane-Role
     - Worker:                  arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Worker-Role
    Operator IAM Roles:
     - arn:aws:iam::<aws_account_id>:role/<cluster_name>-xxxx-openshift-ingress-operator-cloud-credentials
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-cluster-csi-drivers-ebs-cloud-credent
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-machine-api-aws-cloud-credentials
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-cloud-credential-operator-cloud-crede
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-image-registry-installer-cloud-creden
    State:                      ready
    Private:                    No
    Created:                    Oct  1 2021 08:12:25 UTC
    Details Page:               https://console.redhat.com/openshift/details/s/<subscription_id>
    OIDC Endpoint URL:          https://rh-oidc.s3.<aws_region>.amazonaws.com/<cluster_id>

    The following State field changes are listed in the output as the cluster installation progresses:

    • waiting (Waiting for OIDC configuration)

    • pending (Preparing account)

    • installing (DNS setup in progress)

    • installing

    • ready

      If installation fails or the State field does not change to ready after about 40 minutes, check the installation troubleshooting documentation for more details.

  7. Track the progress of the cluster creation by watching the OpenShift installer logs:

    $ rosa logs install --cluster <cluster_name|cluster_id> --watch (1)
    1 Specify the --watch flag to watch for new log messages as the installation progresses. This argument is optional.

Additional resources