×

Create a Red Hat OpenShift Service on AWS (ROSA) cluster with the AWS Security Token Service (STS) using customizations. You can deploy your cluster by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa).

With the procedures in this document, you can also choose between the auto and manual modes when creating the required AWS Identity and Access Management (IAM) resources.

Understanding the auto and manual deployment modes

When installing a Red Hat OpenShift Service on AWS (ROSA) cluster that uses the AWS Security Token Service (STS), you can choose between the auto and manual ROSA CLI (rosa) modes to create the required AWS Identity and Access Management (IAM) resources.

auto mode

With this mode, rosa immediately creates the required IAM roles and policies, and an OpenID Connect (OIDC) provider in your AWS account.

manual mode

With this mode, rosa outputs the aws commands needed to create the IAM resources. The corresponding policy JSON files are also saved to the current directory. By using manual mode, you can review the generated aws commands before running them manually. manual mode also enables you to pass the commands to another administrator or group in your organization so that they can create the resources.

If you opt to use manual mode, the cluster installation waits until you create the cluster-specific Operator roles and OIDC provider manually. After you create the resources, the installation proceeds. For more information, see Creating the Operator roles and OIDC provider using OpenShift Cluster Manager.

For more information about the AWS IAM resources required to install ROSA with STS, see About IAM resources for clusters that use STS.

Creating the Operator roles and OIDC provider using OpenShift Cluster Manager

If you use Red Hat OpenShift Cluster Manager to install your cluster and opt to create the required AWS IAM Operator roles and the OIDC provider using manual mode, you are prompted to select one of the following methods to install the resources. The options are provided to enable you to choose a resource creation method that suits the needs of your organization:

AWS CLI (aws)

With this method, you can download and extract an archive file that contains the aws commands and policy files required to create the IAM resources. Run the provided CLI commands from the directory that contains the policy files to create the Operator roles and the OIDC provider.

ROSA CLI (rosa)

You can run the commands provided by this method to create the Operator roles and the OIDC provider for your cluster using rosa.

If you use auto mode, OpenShift Cluster Manager creates the Operator roles and the OIDC provider automatically, using the permissions provided through the OpenShift Cluster Manager IAM role. To use this feature, you must apply admin privileges to the role.

Understanding AWS account association

Before you can use the Red Hat OpenShift Cluster Manager Hybrid Cloud Console to create Red Hat OpenShift Service on AWS (ROSA) clusters that use the AWS Security Token Service (STS), you must associate your AWS account with your Red Hat organization. You can associate your account by creating and linking the following IAM roles.

OpenShift Cluster Manager role

Create an OpenShift Cluster Manager IAM role and link it to your Red Hat organization.

You can apply basic or administrative permissions to the OpenShift Cluster Manager role. The basic permissions enable cluster maintenance using the OpenShift Cluster Manager Hybrid Cloud Console. The administrative permissions enable automatic deployment of the cluster-specific Operator roles and the OpenID Connect (OIDC) provider using the OpenShift Cluster Manager Hybrid Cloud Console.

User role

Create a user IAM role and link it to your Red Hat user account. The Red Hat user account must exist in the Red Hat organization that is linked to your OpenShift Cluster Manager role.

The user role is used by Red Hat to verify your AWS identity when you use the OpenShift Cluster Manager Hybrid Cloud Console to install a cluster and the required STS resources.

Additional resources

ARN path customization for IAM roles and policies

When you create the AWS IAM roles and policies required for Red Hat OpenShift Service on AWS (ROSA) clusters that use the AWS Security Token Service (STS), you can specify custom Amazon Resource Name (ARN) paths. This enables you to use role and policy ARN paths that meet the security requirements of your organization.

You can specify custom ARN paths when you create your OCM role, user role, and account-wide roles and policies.

If you define a custom ARN path when you create a set of account-wide roles and policies, the same path is applied to all of the roles and policies in the set. The following example shows the ARNs for a set of account-wide roles and policies. In the example, the ARNs use the custom path /test/path/dev/ and the custom role prefix test-env:

  • arn:aws:iam::<account_id>:role/test/path/dev/test-env-Worker-Role

  • arn:aws:iam::<account_id>:role/test/path/dev/test-env-Support-Role

  • arn:aws:iam::<account_id>:role/test/path/dev/test-env-Installer-Role

  • arn:aws:iam::<account_id>:role/test/path/dev/test-env-ControlPlane-Role

  • arn:aws:iam::<account_id>:policy/test/path/dev/test-env-Worker-Role-Policy

  • arn:aws:iam::<account_id>:policy/test/path/dev/test-env-Support-Role-Policy

  • arn:aws:iam::<account_id>:policy/test/path/dev/test-env-Installer-Role-Policy

  • arn:aws:iam::<account_id>:policy/test/path/dev/test-env-ControlPlane-Role-Policy

When you create the cluster-specific Operator roles, the ARN path for the relevant account-wide installer role is automatically detected and applied to the Operator roles.

For more information about ARN paths, see Amazon Resource Names (ARNs) in the AWS documentation.

Additional resources

Support considerations for ROSA clusters with STS

The supported way of creating a Red Hat OpenShift Service on AWS cluster that uses the AWS Security Token Service (STS) is by using the steps described in this product documentation.

You can use manual mode with the Red Hat OpenShift Service on AWS CLI (rosa) to generate the AWS Identity and Access Management (IAM) policy files and aws commands that are required to install the STS resources.

The files and aws commands are generated for review purposes only and must not be modified in any way. Red Hat cannot provide support for ROSA clusters that have been deployed by using modified versions of the policy files or aws commands.

Creating a cluster using customizations

Deploy a Red Hat OpenShift Service on AWS (ROSA) with AWS Security Token Service (STS) cluster with a configuration that suits the needs of your environment. You can deploy your cluster with customizations by using Red Hat OpenShift Cluster Manager or the ROSA CLI (rosa).

Creating a cluster with customizations by using OpenShift Cluster Manager

When you create a Red Hat OpenShift Service on AWS (ROSA) cluster that uses the AWS Security Token Service (STS), you can customize your installation interactively by using Red Hat OpenShift Cluster Manager.

Only public and AWS PrivateLink clusters are supported with STS. Regular private clusters (non-PrivateLink) are not available for use with STS.

AWS Shared VPCs are not currently supported for ROSA installations.

Prerequisites
  • You have completed the AWS prerequisites for ROSA with STS.

  • You have available AWS service quotas.

  • You have enabled the ROSA service in the AWS Console.

  • You have installed and configured the latest ROSA CLI (rosa) on your installation host.

    To successfully install ROSA clusters, use the latest version of the ROSA CLI.

  • You have verified that the AWS Elastic Load Balancing (ELB) service role exists in your AWS account.

  • If you are configuring a cluster-wide proxy, you have verified that the proxy is accessible from the VPC that the cluster is being installed into. The proxy must also be accessible from the private subnets of the VPC.

Procedure
  1. Navigate to OpenShift Cluster Manager Hybrid Cloud Console and select Create cluster.

  2. On the Create an OpenShift cluster page, select Create cluster in the Red Hat OpenShift Service on AWS (ROSA) row.

  3. If an AWS account is automatically detected, the account ID is listed in the Associated AWS accounts drop-down menu. If no AWS accounts are automatically detected, click Select an accountAssociate AWS account and follow these steps:

    1. On the Authenticate page, click the copy button next to the rosa login command. The command includes your OpenShift Cluster Manager API login token.

      You can also load your API token on the OpenShift Cluster Manager API Token page on OpenShift Cluster Manager.

    2. Run the copied command in the CLI to log in to your ROSA account.

      $ rosa login --token=<api_login_token> (1)
      1 Replace <api_login_token> with the token that is provided in the copied command.
      Example output
      I: Logged in as '<username>' on 'https://api.openshift.com'
    3. On the Authenticate page in OpenShift Cluster Manager, click Next.

    4. On the OCM role page, click the copy button next to the Basic OCM role or the Admin OCM role commands.

      The basic role enables OpenShift Cluster Manager to detect the AWS IAM roles and policies required by ROSA. The admin role also enables the detection of the roles and policies. In addition, the admin role enables automatic deployment of the cluster-specific Operator roles and the OpenID Connect (OIDC) provider by using OpenShift Cluster Manager.

    5. Run the copied command in the CLI and follow the prompts to create the OpenShift Cluster Manager IAM role. The following example creates a basic OpenShift Cluster Manager IAM role using the default options:

      $ rosa create ocm-role
      Example output
      I: Creating ocm role
      ? Role prefix: ManagedOpenShift (1)
      ? Enable admin capabilities for the OCM role (optional): No (2)
      ? Permissions boundary ARN (optional):  (3)
      ? Role Path (optional): (4)
      ? Role creation mode: auto (5)
      I: Creating role using 'arn:aws:iam::<aws_account_id>:user/<aws_username>'
      ? Create the 'ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>' role? Yes
      I: Created role 'ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>' with ARN 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>'
      I: Linking OCM role
      ? OCM Role ARN: arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>
      ? Link the 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>' role with organization '<red_hat_organization_id>'? Yes (6)
      I: Successfully linked role-arn 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>' with organization account '<red_hat_organization_id>'
      1 Specify the prefix to include in the OCM IAM role name. The default is ManagedOpenShift. You can create only one OCM role per AWS account for your Red Hat organization.
      2 Enable the admin OpenShift Cluster Manager IAM role, which is equivalent to specifying the --admin argument. The admin role is required if you want to use Auto mode to automatically provision the cluster-specific Operator roles and the OIDC provider by using OpenShift Cluster Manager.
      3 Optional: Specify a permissions boundary Amazon Resource Name (ARN) for the role. For more information, see Permissions boundaries for IAM entities in the AWS documentation.
      4 Specify a custom ARN path for your OCM role. The path must contain alphanumeric characters only and start and end with /, for example /test/path/dev/. For more information, see ARN path customization for IAM roles and policies.
      5 Select the role creation mode. You can use auto mode to automatically create the OpenShift Cluster Manager IAM role and link it to your Red Hat organization account. In manual mode, the rosa CLI generates the aws commands needed to create and link the role. In manual mode, the corresponding policy JSON files are also saved to the current directory. manual mode enables you to review the details before running the aws commands manually.
      6 Link the OpenShift Cluster Manager IAM role to your Red Hat organization account.
    6. If you opted not to link the OpenShift Cluster Manager IAM role to your Red Hat organization account in the preceding command, copy the rosa link command from the OpenShift Cluster Manager OCM role page and run it:

      $ rosa link ocm-role <arn> (1)
      1 Replace <arn> with the ARN of the OpenShift Cluster Manager IAM role that is included in the output of the preceding command.
    7. Select Next on the OpenShift Cluster Manager OCM role page.

    8. On the User role page, click the copy button for the User role command and run the command in the CLI. Red Hat uses the user role to verify your AWS identity when you install a cluster and the required resources with OpenShift Cluster Manager.

      Follow the prompts to create the user role:

      $ rosa create user-role
      Example output
      I: Creating User role
      ? Role prefix: ManagedOpenShift (1)
      ? Permissions boundary ARN (optional): (2)
      ? Role Path (optional): [? for help] (3)
      ? Role creation mode: auto (4)
      I: Creating ocm user role using 'arn:aws:iam::<aws_account_id>:user/<aws_username>'
      ? Create the 'ManagedOpenShift-User-<red_hat_username>-Role' role? Yes
      I: Created role 'ManagedOpenShift-User-<red_hat_username>-Role' with ARN 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<red_hat_username>-Role'
      I: Linking User role
      ? User Role ARN: arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<red_hat_username>-Role
      ? Link the 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<red_hat_username>-Role' role with account '<red_hat_user_account_id>'? Yes (5)
      I: Successfully linked role ARN 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<red_hat_username>-Role' with account '<red_hat_user_account_id>'
      1 Specify the prefix to include in the user role name. The default is ManagedOpenShift.
      2 Optional: Specify a permissions boundary Amazon Resource Name (ARN) for the role. For more information, see Permissions boundaries for IAM entities in the AWS documentation.
      3 Specify a custom ARN path for your user role. The path must contain alphanumeric characters only and start and end with /, for example /test/path/dev/. For more information, see ARN path customization for IAM roles and policies.
      4 Select the role creation mode. You can use auto mode to automatically create the user role and link it to your OpenShift Cluster Manager user account. In manual mode, the rosa CLI generates the aws commands needed to create and link the role. In manual mode, the corresponding policy JSON files are also saved to the current directory. manual mode enables you to review the details before running the aws commands manually.
      5 Link the user role to your OpenShift Cluster Manager user account.
    9. If you opted not to link the user role to your OpenShift Cluster Manager user account in the preceding command, copy the rosa link command from the OpenShift Cluster Manager User role page and run it:

      $ rosa link user-role <arn> (1)
      1 Replace <arn> with the ARN of the user role that is included in the output of the preceding command.
    10. On the OpenShift Cluster Manager User role page, click Ok.

    11. Verify that the AWS account ID is listed in the Associated AWS accounts drop-down menu on the Accounts and roles page.

    12. If the required account roles do not exist, a notification is provided stating that Some account roles ARNs were not detected. You can create the AWS account-wide roles and policies, including the Operator policies, by clicking the copy buffer next to the rosa create account-roles command and running the command in the CLI:

      $ rosa create account-roles
      Example output
      I: Logged in as '<red_hat_username>' on 'https://api.openshift.com'
      I: Validating AWS credentials...
      I: AWS credentials are valid!
      I: Validating AWS quota...
      I: AWS quota ok. If cluster installation fails, validate actual AWS resource usage against https://docs.openshift.com/rosa/rosa_getting_started/rosa-required-aws-service-quotas.html
      I: Verifying whether OpenShift command-line tool is available...
      I: Current OpenShift Client Version: 4.11.6
      I: Creating account roles
      ? Role prefix: ManagedOpenShift (1)
      ? Permissions boundary ARN (optional): (2)
      ? Path (optional): [? for help] (3)
      ? Role creation mode: auto (4)
      I: Creating roles using 'arn:aws:iam::<aws_account_number>:user/<aws_username>'
      ? Create the 'ManagedOpenShift-Installer-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-Installer-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-Installer-Role'
      ? Create the 'ManagedOpenShift-ControlPlane-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-ControlPlane-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-ControlPlane-Role'
      ? Create the 'ManagedOpenShift-Worker-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-Worker-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-Worker-Role'
      ? Create the 'ManagedOpenShift-Support-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-Support-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-Support-Role'
      I: To create a cluster with these roles, run the following command:
      rosa create cluster --sts
      1 Specify the prefix to include in the OpenShift Cluster Manager IAM role name. The default is ManagedOpenShift.

      You must specify an account-wide role prefix that is unique across your AWS account, even if you use a custom ARN path for your account roles.

      2 Optional: Specify a permissions boundary Amazon Resource Name (ARN) for the role. For more information, see Permissions boundaries for IAM entities in the AWS documentation.
      3 Specify a custom ARN path for your account-wide roles. The path must contain alphanumeric characters only and start and end with /, for example /test/path/dev/. For more information, see ARN path customization for IAM roles and policies.
      4 Select the role creation mode. You can use auto mode to automatically create the account wide roles and policies. In manual mode, the rosa CLI generates the aws commands needed to create the roles and policies. In manual mode, the corresponding policy JSON files are also saved to the current directory. manual mode enables you to review the details before running the aws commands manually.
      5 Creates the account-wide installer, control plane, worker and support roles and corresponding IAM policies. For more information, see Account-wide IAM role and policy reference.

      In this step, the ROSA CLI also automatically creates the account-wide Operator IAM policies that are used by the cluster-specific Operator policies to permit the ROSA cluster Operators to carry out core OpenShift functionality. For more information, see Account-wide IAM role and policy reference.

    13. On the Accounts and roles page, click Refresh ARNs and verify that the installer, support, worker, and control plane account role ARNs are listed.

      If you have more than one set of account roles in your AWS account for your cluster version, a drop-down list of Installer role ARNs is provided. Select the ARN for the installer role that you want to use with your cluster. The cluster uses the account-wide roles and policies that relate to the selected installer role.

  4. Click Next.

    If the Accounts and roles page was refreshed, you might need to select the checkbox again to acknowledge that you have read and completed all of the prerequisites.

  5. On the Cluster details page, provide a name for your cluster and specify the cluster details:

    1. Add a Cluster name.

    2. Select a cluster version from the Version drop-down menu.

    3. Select a cloud provider region from the Region drop-down menu.

    4. Select a Single zone or Multi-zone configuration.

    5. Leave Enable user workload monitoring selected to monitor your own projects in isolation from Red Hat Site Reliability Engineer (SRE) platform metrics. This option is enabled by default.

    6. Optional: Select Enable additional etcd encryption if you require etcd key value encryption. With this option, the etcd key values are encrypted, but not the keys. This option is in addition to the control plane storage encryption that encrypts the etcd volumes in Red Hat OpenShift Service on AWS clusters by default.

      By enabling etcd encryption for the key values in etcd, you will incur a performance overhead of approximately 20%. The overhead is a result of introducing this second layer of encryption, in addition to the default control plane storage encryption that encrypts the etcd volumes. Consider enabling etcd encryption only if you specifically require it for your use case.

    7. Optional: Select Encrypt persistent volumes with customer keys if you want to provide your own AWS Key Management Service (KMS) key Amazon Resource Name (ARN). The key is used for encryption of persistent volumes in your cluster.

      Only persistent volumes (PVs) created from the default storage class are encrypted by default.

      PVs created by using any other storage class are only encrypted if the storage class is configured to be encrypted.

    8. Click Next.

  6. On the Default machine pool page, select a Compute node instance type.

    After your cluster is created, you can change the number of compute nodes in your cluster, but you cannot change the compute node instance type in the default machine pool. The number and types of nodes available to you depend on whether you use single or multiple availability zones. They also depend on what is enabled and available in your AWS account and the selected region.

  7. Optional: Configure autoscaling for the default machine pool:

    1. Select Enable autoscaling to automatically scale the number of machines in your default machine pool to meet the deployment needs.

    2. Set the minimum and maximum node count limits for autoscaling. The cluster autoscaler does not reduce or increase the default machine pool node count beyond the limits that you specify.

      • If you deployed your cluster using a single availability zone, set the Minimum node count and Maximum node count. This defines the minimum and maximum compute node limits in the availability zone.

      • If you deployed your cluster using multiple availability zones, set the Minimum nodes per zone and Maximum nodes per zone. This defines the minimum and maximum compute node limits per zone.

      Alternatively, you can set your autoscaling preferences for the default machine pool after the machine pool is created.

  8. If you did not enable autoscaling, select a compute node count for your default machine pool:

    • If you deployed your cluster using a single availability zone, select a Compute node count from the drop-down menu. This defines the number of compute nodes to provision to the machine pool for the zone.

    • If you deployed your cluster using multiple availability zones, select a Compute node count (per zone) from the drop-down menu. This defines the number of compute nodes to provision to the machine pool per zone.

  9. Optional: Expand Edit node labels to add labels to your nodes. Click Add label to add more node labels and select Next.

  10. In the Cluster privacy section of the Network configuration page, select Public or Private to use either public or private API endpoints and application routes for your cluster.

    The API endpoint cannot be changed between public and private after your cluster is created.

    Public API endpoint

    Select Public if you do not want to restrict access to your cluster. You can access the Kubernetes API endpoint and application routes from the internet.

    Private API endpoint

    Select Private if you want to restrict network access to to your cluster. The Kubernetes API endpoint and application routes are accessible from direct private connections only.

    If you are using private API endpoints, you cannot access your cluster until you update the network settings in your cloud provider account.

  11. Optional: If you opted to use public API endpoints, by default a new VPC is created for your cluster. If you want to install your cluster in an existing VPC instead, select Install into an existing VPC.

    If you opted to use private API endpoints, you must use an existing VPC and PrivateLink and the Install into an existing VPC and Use a PrivateLink options are automatically selected. With these options, the Red Hat Site Reliability Engineering (SRE) team can connect to the cluster to assist with support by using only AWS PrivateLink endpoints.

  12. Optional: If you are installing your cluster into an existing VPC, select Configure a cluster-wide proxy to enable an HTTP or HTTPS proxy to deny direct access to the internet from your cluster.

  13. Click Next.

  14. If you opted to install the cluster in an existing AWS VPC, provide your Virtual Private Cloud (VPC) subnet settings.

    You must ensure that your VPC is configured with a public and a private subnet for each availability zone that you want the cluster installed into. If you opted to use PrivateLink, only private subnets are required.

  15. If you opted to configure a cluster-wide proxy, provide your proxy configuration details on the Cluster-wide proxy page:

    1. Enter a value in at least one of the following fields:

      • Specify a valid HTTP proxy URL.

      • Specify a valid HTTPS proxy URL.

      • In the Additional trust bundle field, provide a PEM encoded X.509 certificate bundle. The bundle is added to the trusted certificate store for the cluster nodes. An additional trust bundle file is required unless the identity certificate for the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle.

        If you use an MITM transparent proxy network that does not require additional proxy configuration but requires additional certificate authorities (CAs), you must provide the MITM CA certificate.

        If you upload an additional trust bundle file without specifying an HTTP or HTTPS proxy URL, the bundle is set on the cluster but is not configured to be used with the proxy.

    2. Click Next.

    For more information about configuring a proxy with Red Hat OpenShift Service on AWS, see Configuring a cluster-wide proxy.

  16. In the CIDR ranges dialog, configure custom classless inter-domain routing (CIDR) ranges or use the defaults that are provided and click Next.

    If you are installing into a VPC, the Machine CIDR range must match the VPC subnets.

    CIDR configurations cannot be changed later. Confirm your selections with your network administrator before proceeding.

  17. Under the Cluster roles and policies page, select your preferred cluster-specific Operator IAM role and OIDC provider creation mode.

    With Manual mode, you can use either rosa CLI commands or aws CLI commands to generate the required Operator roles and OIDC provider for your cluster. Manual mode enables you to review the details before using your preferred option to create the IAM resources manually and complete your cluster installation.

    Alternatively, you can use Auto mode to automatically create the Operator roles and OIDC provider. To enable Auto mode, the OpenShift Cluster Manager IAM role must have administrator capabilities.

    If you specified custom ARN paths when you created the associated account-wide roles, the custom path is automatically detected and applied to the Operator roles. The custom ARN path is applied when the Operator roles are created by using either Manual or Auto mode.

  18. Optional: Specify a Custom operator roles prefix for your cluster-specific Operator IAM roles.

    By default, the cluster-specific Operator role names are prefixed with the cluster name and random 4-digit hash. You can optionally specify a custom prefix to replace <cluster_name>-<hash> in the role names. The prefix is applied when you create the cluster-specific Operator IAM roles. For information about the prefix, see About custom Operator IAM role prefixes.

  19. Select Next.

  20. On the Cluster update strategy page, configure your update preferences:

    1. Choose a cluster update method:

      • Select Individual updates if you want to schedule each update individually. This is the default option.

      • Select Recurring updates to update your cluster on your preferred day and start time, when updates are available.

        Even when you opt for recurring updates, you must update the account-wide and cluster-specific IAM resources before you upgrade your cluster between minor releases.

        You can review the end-of-life dates in the update life cycle documentation for Red Hat OpenShift Service on AWS. For more information, see Red Hat OpenShift Service on AWS update life cycle.

    2. If you opted for recurring updates, select a preferred day of the week and upgrade start time in UTC from the drop-down menus.

    3. Optional: You can set a grace period for Node draining during cluster upgrades. A 1 hour grace period is set by default.

    4. Click Next.

      In the event of critical security concerns that significantly impact the security or stability of a cluster, Red Hat Site Reliability Engineering (SRE) might schedule automatic updates to the latest z-stream version that is not impacted. The updates are applied within 48 hours after customer notifications are provided. For a description of the critical impact security rating, see Understanding Red Hat security ratings.

  21. Review the summary of your selections and click Create cluster to start the cluster installation.

  22. If you opted to use Manual mode, create the cluster-specific Operator roles and OIDC provider manually to continue the installation:

    1. In the Action required to continue installation dialog, select either the AWS CLI or ROSA CLI tab and manually create the resources:

      • If you opted to use the AWS CLI method, click Download .zip, save the file, and then extract the AWS CLI command and policy files. Then, run the provided aws commands in the CLI.

        You must run the aws commands in the directory that contains the policy files.

      • If you opted to use the ROSA CLI method, click the copy button next to the rosa create commands and run them in the CLI.

        If you specified custom ARN paths when you created the associated account-wide roles, the custom path is automatically detected and applied to the Operator roles when you create them by using these manual methods.

    2. In the Action required to continue installation dialog, click x to return to the Overview page for your cluster.

    3. Verify that the cluster Status in the Details section of the Overview page for your cluster has changed from Waiting to Installing. There might be a short delay of approximately two minutes before the status changes.

    If you opted to use Auto mode, OpenShift Cluster Manager creates the Operator roles and the OIDC provider automatically.

Verification
  • You can monitor the progress of the installation in the Overview page for your cluster. You can view the installation logs on the same page. Your cluster is ready when the Status in the Details section of the page is listed as Ready.

    If the installation fails or the cluster State does not change to Ready after about 40 minutes, check the installation troubleshooting documentation for details. For more information, see Troubleshooting installations. For steps to contact Red Hat Support for assistance, see Getting support for Red Hat OpenShift Service on AWS.

Creating a cluster with customizations using the CLI

When you create a Red Hat OpenShift Service on AWS (ROSA) cluster that uses the AWS Security Token Service (STS), you can customize your installation interactively.

When you run rosa create cluster --interactive at cluster creation time, you are presented with a series of interactive prompts that enable you to customize your deployment. For more information, see Interactive cluster creation mode reference.

After a cluster installation using the interactive mode completes, a single command is provided in the output that enables you to deploy further clusters using the same custom configuration.

Only public and AWS PrivateLink clusters are supported with STS. Regular private clusters (non-PrivateLink) are not available for use with STS.

AWS Shared VPCs are not currently supported for ROSA installations.

Prerequisites
  • You have completed the AWS prerequisites for ROSA with STS.

  • You have available AWS service quotas.

  • You have enabled the ROSA service in the AWS Console.

  • You have installed and configured the latest ROSA (rosa) and AWS (aws) CLIs on your installation host.

    To successfully install ROSA clusters, use latest version of the ROSA CLI.

  • If you are using a customer-managed AWS Key Management Service (KMS) key for encryption, you have created a symmetric KMS key and you have the key ID and Amazon Resource Name (ARN). For more information about creating AWS KMS keys, see the AWS documentation.

Procedure
  1. Create the required account-wide roles and policies, including the Operator policies:

    1. Generate the IAM policy JSON files in the current working directory and output the aws CLI commands for review:

      $ rosa create account-roles --interactive \ (1)
                                  --mode manual (2)
      
      1 interactive mode enables you to specify configuration options at the interactive prompts. For more information, see Interactive cluster creation mode reference.
      2 manual mode generates the aws CLI commands and JSON files needed to create the account-wide roles and policies. After review, you must run the commands manually to create the resources.
      Example output
      I: Logged in as '<red_hat_username>' on 'https://api.openshift.com'
      I: Validating AWS credentials...
      I: AWS credentials are valid!
      I: Validating AWS quota...
      I: AWS quota ok. If cluster installation fails, validate actual AWS resource usage against https://docs.openshift.com/rosa/rosa_getting_started/rosa-required-aws-service-quotas.html
      I: Verifying whether OpenShift command-line tool is available...
      I: Current OpenShift Client Version: 4.11.6
      I: Creating account roles
      ? Role prefix: ManagedOpenShift (1)
      ? Permissions boundary ARN (optional): (2)
      ? Path (optional): [? for help] (3)
      ? Role creation mode: auto (4)
      I: Creating roles using 'arn:aws:iam::<aws_account_number>:user/<aws_username>'
      ? Create the 'ManagedOpenShift-Installer-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-Installer-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-Installer-Role'
      ? Create the 'ManagedOpenShift-ControlPlane-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-ControlPlane-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-ControlPlane-Role'
      ? Create the 'ManagedOpenShift-Worker-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-Worker-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-Worker-Role'
      ? Create the 'ManagedOpenShift-Support-Role' role? Yes (5)
      I: Created role 'ManagedOpenShift-Support-Role' with ARN 'arn:aws:iam::<aws_account_number>:role/ManagedOpenShift-Support-Role'
      I: To create a cluster with these roles, run the following command:
      rosa create cluster --sts
      1 Specify the prefix to include in the OpenShift Cluster Manager IAM role name. The default is ManagedOpenShift.

      You must specify an account-wide role prefix that is unique across your AWS account, even if you use a custom ARN path for your account roles.

      2 Optional: Specifies a permissions boundary Amazon Resource Name (ARN) for the role. For more information, see Permissions boundaries for IAM entities in the AWS documentation.
      3 Specify a custom ARN path for your account-wide roles. The path must contain alphanumeric characters only and start and end with /, for example /test/path/dev/. For more information, see ARN path customization for IAM roles and policies.
      4 Select the role creation mode. You can use auto mode to automatically create the account wide roles and policies. In manual mode, the rosa CLI generates the aws commands needed to create the roles and policies. In manual mode, the corresponding policy JSON files are also saved to the current directory. manual mode enables you to review the details before running the aws commands manually.
      5 Creates the account-wide installer, control plane, worker and support roles and corresponding IAM policies. For more information, see Account-wide IAM role and policy reference.

      In this step, the ROSA CLI also automatically creates the account-wide Operator IAM policies that are used by the cluster-specific Operator policies to permit the ROSA cluster Operators to carry out core OpenShift functionality. For more information, see Account-wide IAM role and policy reference.

    2. After review, run the aws commands manually to create the roles and policies. Alternatively, you can run the preceding command using --mode auto to run the aws commands immediately.

  2. Optional: If you are using your own AWS KMS key to encrypt the control plane, infrastructure, worker node root volumes, and persistent volumes (PVs), add the ARN for the account-wide installer role to your KMS key policy.

    Only persistent volumes (PVs) created from the default storage class are encrypted with this specific key.

    PVs created by using any other storage class are still encrypted, but the PVs are not encrypted with this key unless the storage class is specifically configured to use this key.

    1. Save the key policy for your KMS key to a file on your local machine. The following example saves the output to kms-key-policy.json in the current working directory:

      $ aws kms get-key-policy --key-id <key_id_or_arn> --policy-name default --output text > kms-key-policy.json (1)
      1 Replace <key_id_or_arn> with the ID or ARN of your KMS key.
    2. Add the ARN for the account-wide installer role that you created in the preceding step to the Statement.Principal.AWS section in the file. In the following example, the ARN for the default ManagedOpenShift-Installer-Role role is added:

      {
          "Version": "2012-10-17",
          "Id": "key-rosa-policy-1",
          "Statement": [
              {
                  "Sid": "Enable IAM User Permissions",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::<aws-account-id>:root"
                  },
                  "Action": "kms:*",
                  "Resource": "*"
              },
              {
                  "Sid": "Allow ROSA use of the key",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": [
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-Support-Role", (1)
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-Installer-Role",
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-Worker-Role",
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-ControlPlane-Role"
                      ]
                  },
                  "Action": [
                      "kms:Encrypt",
                      "kms:Decrypt",
                      "kms:ReEncrypt*",
                      "kms:GenerateDataKey*",
                      "kms:DescribeKey"
                  ],
                  "Resource": "*"
              },
              {
                  "Sid": "Allow attachment of persistent resources",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": [
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-Support-Role", (1)
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-Installer-Role",
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-Worker-Role",
                          "arn:aws:iam::<aws-account-id>:role/ManagedOpenShift-ControlPlane-Role"
                      ]
                  },
                  "Action": [
                      "kms:CreateGrant",
                      "kms:ListGrants",
                      "kms:RevokeGrant"
                  ],
                  "Resource": "*",
                  "Condition": {
                      "Bool": {
                          "kms:GrantIsForAWSResource": "true"
                      }
                  }
              }
          ]
      }
      1 You must specify the ARN for the account-wide role that will be used when you create the ROSA cluster. The ARNs listed in the section must be comma-separated.
    3. Apply the changes to your KMS key policy:

      $ aws kms put-key-policy --key-id <key_id_or_arn> \ (1)
          --policy file://kms-key-policy.json \ (2)
          --policy-name default
      1 Replace <key_id_or_arn> with the ID or ARN of your KMS key.
      2 You must include the file:// prefix when referencing a key policy in a local file.

      You can reference the ARN of your KMS key when you create the cluster in the next step.

  3. Create a cluster with STS using custom installation options. You can use the --interactive mode to interactively specify custom settings:

    $ rosa create cluster --interactive --sts
    Example output
    I: Interactive mode enabled.
    Any optional fields can be left empty and a default will be selected.
    ? Cluster name: <cluster_name>
    ? OpenShift version: 4.8.9 (1)
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Installer-Role for the Installer role (2)
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-ControlPlane-Role for the ControlPlane role
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Worker-Role for the Worker role
    I: Using arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Support-Role for the Support role
    ? External ID (optional):
    ? Operator roles prefix: <cluster_name>-<random_string> (3)
    ? Multiple availability zones (optional): No (4)
    ? AWS region: us-east-1
    ? PrivateLink cluster (optional): No
    ? Install into an existing VPC (optional): No
    ? Select availability zones (optional): No
    ? Enable Customer Managed key (optional): No (5)
    ? Compute nodes instance type (optional):
    ? Enable autoscaling (optional): No
    ? Compute nodes: 2
    ? Machine CIDR: 10.0.0.0/16
    ? Service CIDR: 172.30.0.0/16
    ? Pod CIDR: 10.128.0.0/14
    ? Host prefix: 23
    ? Encrypt etcd data (optional): No (6)
    ? Disable Workload monitoring (optional): No
    I: Creating cluster '<cluster_name>'
    I: To create this cluster again in the future, you can run:
       rosa create cluster --cluster-name <cluster_name> --role-arn arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Installer-Role --support-role-arn arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Support-Role --master-iam-role arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-ControlPlane-Role --worker-iam-role arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Worker-Role --operator-roles-prefix <cluster_name>-<random_string> --region us-east-1 --version 4.8.9 --compute-nodes 2 --machine-cidr 10.0.0.0/16 --service-cidr 172.30.0.0/16 --pod-cidr 10.128.0.0/14 --host-prefix 23 (7)
    I: To view a list of clusters and their status, run 'rosa list clusters'
    I: Cluster '<cluster_name>' has been created.
    I: Once the cluster is installed you will need to add an Identity Provider before you can login into the cluster. See 'rosa create idp --help' for more information.
    ...
    1 When creating the cluster, the listed OpenShift version options include the major, minor, and patch versions, for example 4.8.9.
    2 If you have more than one set of account roles in your AWS account for your cluster version, an interactive list of options is provided.
    3 Optional: By default, the cluster-specific Operator role names are prefixed with the cluster name and random 4-digit hash. You can optionally specify a custom prefix to replace <cluster_name>-<hash> in the role names. The prefix is applied when you create the cluster-specific Operator IAM roles. For information about the prefix, see Defining an Operator IAM role prefix.

    If you specified custom ARN paths when you created the associated account-wide roles, the custom path is automatically detected. The custom path is applied to the cluster-specific Operator roles when you create them in a later step.

    4 Multiple availability zones are recommended for production workloads. The default is a single availability zone.
    5 Enable this option if you are using your own AWS KMS key to encrypt the control plane, infrastructure, worker node root volumes, and PVs. Specify the ARN for the KMS key that you added to the account-wide role ARN to in the preceding step.

    Only persistent volumes (PVs) created from the default storage class are encrypted with this specific key.

    PVs created by using any other storage class are still encrypted, but the PVs are not encrypted with this key unless the storage class is specifically configured to use this key.

    6 Enable this option only if your use case requires etcd key value encryption in addition to the control plane storage encryption that encrypts the etcd volumes by default. With this option, the etcd key values are encrypted, but not the keys.

    By enabling etcd encryption for the key values in etcd, you will incur a performance overhead of approximately 20%. The overhead is a result of introducing this second layer of encryption, in addition to the default control plane storage encryption that encrypts the etcd volumes. Red Hat recommends that you enable etcd encryption only if you specifically require it for your use case.

    7 The output includes a custom command that you can run to create a cluster with the same configuration in the future.

    As an alternative to using the --interactive mode, you can specify the customization options directly when you run rosa create cluster. Run rosa create cluster --help to view a list of available CLI options.

    You must complete the following steps to create the Operator IAM roles and the OpenID Connect (OIDC) provider to move the state of the cluster to ready.

  4. Create the cluster-specific Operator IAM roles:

    1. Generate the Operator IAM policy JSON files in the current working directory and output the aws CLI commands for review:

      $ rosa create operator-roles --mode manual --cluster <cluster_name|cluster_id> (1)
      1 manual mode generates the aws CLI commands and JSON files needed to create the Operator roles. After review, you must run the commands manually to create the resources.
    2. After review, run the aws commands manually to create the Operator IAM roles and attach the managed Operator policies to them. Alternatively, you can run the preceding command again using --mode auto to run the aws commands immediately.

      A custom prefix is applied to the Operator role names if you specified the prefix in the preceding step.

      If you specified custom ARN paths when you created the associated account-wide roles, the custom path is automatically detected and applied to the Operator roles.

  5. Create the OpenID Connect (OIDC) provider that the cluster Operators use to authenticate:

    $ rosa create oidc-provider --mode auto --cluster <cluster_name|cluster_id> (1)
    1 auto mode immediately runs the aws CLI command that creates the OIDC provider.
  6. Check the status of your cluster:

    $ rosa describe cluster --cluster <cluster_name|cluster_id>
    Example output
    Name:                       <cluster_name>
    ID:                         <cluster_id>
    External ID:                <external_id>
    OpenShift Version:          <version>
    Channel Group:              stable
    DNS:                        <cluster_name>.xxxx.p1.openshiftapps.com
    AWS Account:                <aws_account_id>
    API URL:                    https://api.<cluster_name>.xxxx.p1.openshiftapps.com:6443
    Console URL:                https://console-openshift-console.apps.<cluster_name>.xxxx.p1.openshiftapps.com
    Region:                     <aws_region>
    Multi-AZ:                   false
    Nodes:
     - Master:                  3
     - Infra:                   2
     - Compute:                 2
    Network:
     - Service CIDR:            172.30.0.0/16
     - Machine CIDR:            10.0.0.0/16
     - Pod CIDR:                10.128.0.0/14
     - Host Prefix:             /23
    STS Role ARN:               arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Installer-Role
    Support Role ARN:           arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Support-Role
    Instance IAM Roles:
     - Master:                  arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-ControlPlane-Role
     - Worker:                  arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-Worker-Role
    Operator IAM Roles:
     - arn:aws:iam::<aws_account_id>:role/<cluster_name>-xxxx-openshift-ingress-operator-cloud-credentials
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-cluster-csi-drivers-ebs-cloud-credent
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-machine-api-aws-cloud-credentials
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-cloud-credential-operator-cloud-crede
     - arn:aws:iam::<aws_account_id>:role/<cluster_name-xxxx-openshift-image-registry-installer-cloud-creden
    State:                      ready
    Private:                    No
    Created:                    Oct  1 2021 08:12:25 UTC
    Details Page:               https://console.redhat.com/openshift/details/s/<subscription_id>
    OIDC Endpoint URL:          https://rh-oidc.s3.<aws_region>.amazonaws.com/<cluster_id>

    The following State field changes are listed in the output as the cluster installation progresses:

    • waiting (Waiting for OIDC configuration)

    • pending (Preparing account)

    • installing (DNS setup in progress)

    • installing

    • ready

      If the installation fails or the State field does not change to ready after about 40 minutes, check the installation troubleshooting documentation for details. For more information, see Troubleshooting installations. For steps to contact Red Hat Support for assistance, see Getting support for Red Hat OpenShift Service on AWS.

  7. Track the progress of the cluster creation by watching the OpenShift installer logs:

    $ rosa logs install --cluster <cluster_name|cluster_id> --watch (1)
    1 Specify the --watch flag to watch for new log messages as the installation progresses. This argument is optional.

Additional resources