Red Hat OpenShift Service on AWS (ROSA) provides a model that allows Red Hat to deploy clusters into a customer’s existing Amazon Web Service (AWS) account.

You must ensure that the prerequisites are met before installing ROSA. This requirements document does not apply to AWS Security Token Service (STS). If you are using STS, see the STS-specific requirements.

Deployment Prerequisites

To deploy Red Hat OpenShift Service on AWS (ROSA) into your existing Amazon Web Services (AWS) account, Red Hat requires that several prerequisites are met.

Red Hat recommends the usage of AWS Organizations to manage multiple AWS accounts. The AWS Organizations, managed by the customer, host multiple AWS accounts. There is a root account in the organization that all accounts will refer to in the account hierarchy.

It is a best practice for the ROSA cluster to be hosted in an AWS account within an AWS Organizational Unit. A Service Control Policy (SCP) is created and applied to the AWS Organizational Unit that manages what services the AWS sub-accounts are permitted to access. The SCP applies only to available permissions within a single AWS account for all AWS sub-accounts within the Organizational Unit. It is also possible to apply a SCP to a single AWS account. All other accounts in the customer’s AWS Organizations are managed in whatever manner the customer requires. Red Hat Site Reliability Engineers (SRE) will not have any control over SCPs within AWS Organizations.

Customer Requirements

Red Hat OpenShift Service on AWS (ROSA) clusters must meet several prerequisites before they can be deployed.

In order to create the cluster, the user must be logged in as an IAM user and not an assumed role or STS user.

Account

  • The customer ensures that the AWS limits are sufficient to support Red Hat OpenShift Service on AWS provisioned within the customer’s AWS account.

  • The customer’s AWS account should be in the customer’s AWS Organizations with the applicable Service Control Policy (SCP) applied.

    It is not a requirement that the customer’s account be within the AWS Organizations or for the SCP to be applied, however Red Hat must be able to perform all the actions listed in the SCP without restriction.

  • The customer’s AWS account should not be transferable to Red Hat.

  • The customer may not impose AWS usage restrictions on Red Hat activities. Imposing restrictions will severely hinder Red Hat’s ability to respond to incidents.

  • The customer may deploy native AWS services within the same AWS account.

    Customers are encouraged, but not mandated, to deploy resources in a Virtual Private Cloud (VPC) separate from the VPC hosting Red Hat OpenShift Service on AWS and other Red Hat supported services.

Access requirements

  • To appropriately manage the Red Hat OpenShift Service on AWS service, Red Hat must have the AdministratorAccess policy applied to the administrator role at all times. This requirement does not apply if you are using AWS Security Token Service (STS).

    This policy only provides Red Hat with permissions and capabilities to change resources in the customer-provided AWS account.

  • Red Hat must have AWS console access to the customer-provided AWS account. This access is protected and managed by Red Hat.

  • The customer must not utilize the AWS account to elevate their permissions within the Red Hat OpenShift Service on AWS cluster.

  • Actions available in the rosa CLI utility or OpenShift Cluster Manager (OCM) console must not be directly performed in the customer’s AWS account.

Support requirements

  • Red Hat recommends that the customer have at least Business Support from AWS.

  • Red Hat has authority from the customer to request AWS support on their behalf.

  • Red Hat has authority from the customer to request AWS resource limit increases on the customer’s account.

  • Red Hat manages the restrictions, limitations, expectations, and defaults for all Red Hat OpenShift Service on AWS clusters in the same manner, unless otherwise specified in this requirements section.

Security requirements

  • Volume snapshots will remain within the customer’s AWS account and customer-specified region.

  • Red Hat must have ingress access to EC2 hosts and the API server from allow-listed IP addresses.

  • Red Hat must have egress allowed to forward system and audit logs to a Red Hat managed central logging stack.

Required customer procedure

Complete these steps before deploying Red Hat OpenShift Service on AWS (ROSA).

Procedure
  1. If you, as the customer, are utilizing AWS Organizations, then you must use an AWS account within your organization or create a new one.

  2. To ensure that Red Hat can perform necessary actions, you must either create a Service Control Policy (SCP) or ensure that none is applied to the AWS account.

  3. Attach the SCP to the AWS account.

  4. Follow the ROSA procedures for setting up the environment.

Minimum required Service Control Policy (SCP)

Service Control Policy (SCP) management is the responsibility of the customer. These policies are maintained in the AWS Organizations and control what services are available within the attached AWS accounts.

The minimum SPC requirement does not apply when using AWS security token service (STS). For more information about STS, see AWS prerequisites for ROSA with STS.

Service Actions Effect

Required

Amazon EC2

All

Allow

Amazon EC2 Auto Scaling

All

Allow

Amazon S3

All

Allow

Identity And Access Management

All

Allow

Elastic Load Balancing

All

Allow

Elastic Load Balancing V2

All

Allow

Amazon CloudWatch

All

Allow

Amazon CloudWatch Events

All

Allow

Amazon CloudWatch Logs

All

Allow

AWS Support

All

Allow

AWS Key Management Service

All

Allow

AWS Security Token Service

All

Allow

AWS Resource Tagging

All

Allow

AWS Route53 DNS

All

Allow

AWS Service Quotas

ListServices

GetRequestedServiceQuotaChange

GetServiceQuota

RequestServiceQuotaIncrease

ListServiceQuotas

Allow

Optional

AWS Billing

ViewAccount

Viewbilling

ViewUsage

Allow

AWS Cost and Usage Report

All

Allow

AWS Cost Explorer Services

All

Allow

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "events:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "support:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sts:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "tag:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "servicequotas:ListServices",
                "servicequotas:GetRequestedServiceQuotaChange",
                "servicequotas:GetServiceQuota",
                "servicequotas:RequestServiceQuotaIncrease",
                "servicequotas:ListServiceQuotas"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Red Hat managed IAM references for AWS

Red Hat is responsible for creating and managing the following Amazon Web Services (AWS) resources: IAM policies, IAM users, and IAM roles.

IAM Policies

IAM policies are subject to modification as the capabilities of Red Hat OpenShift Service on AWS change.

  • The AdministratorAccess policy is used by the administration role. This policy provides Red Hat the access necessary to administer the Red Hat OpenShift Service on AWS (ROSA) cluster in the customer’s AWS account.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "*",
                "Resource": "*",
                "Effect": "Allow"
            }
        ]
    }

IAM users

The osdManagedAdmin user is created immediately after installing ROSA into the customer’s AWS account.

Provisioned AWS Infrastructure

This is an overview of the provisioned Amazon Web Services (AWS) components on a deployed Red Hat OpenShift Service on AWS (ROSA) cluster. For a more detailed listing of all provisioned AWS components, see the OpenShift Container Platform documentation.

EC2 instances

AWS EC2 instances are required for deploying the control plane and data plane functions of ROSA in the AWS public cloud.

Instance types can vary for control plane and infrastructure nodes, depending on the worker node count.

  • Three m5.xlarge minimum (control plane nodes)

  • Two r5.xlarge minimum (infrastructure nodes)

  • Two m5.xlarge minimum but highly variable (worker nodes)

Elastic Block Storage storage

Amazon EBS block storage is used for both local node storage and persistent volume storage.

Volume requirements for each EC2 instance:

  • Control Plane Volume

    • Size: 350GB

    • Type: io1

    • Input/Output Operations Per Second: 1000

  • Infrastructure Volume

    • Size: 300GB

    • Type: gp2

    • Input/Output Operations Per Second: 100

  • Worker Volume

    • Size: 300GB

    • Type: gp2

    • Input/Output Operations Per Second: 100

Elastic load balancers

Up to two Network Elastic Load Balancers (ELBs) for API and up to two Classic ELBs for application router. For more information, see the ELB documentation for AWS.

S3 storage

The image registry and Elastic Block Store (EBS) volume snapshots are backed by AWS S3 storage. Pruning of resources is performed regularly to optimize S3 usage and cluster performance.

Two buckets are required with a typical size of 2TB each.

VPC

Customers should expect to see one VPC per cluster. Additionally, the VPC will need the following configurations:

  • Subnets: Two subnets for a cluster with a single availability zone, or six subnets for a cluster with multiple availability zones.

  • Router tables: One router table per private subnet, and one additional table per cluster.

  • Internet gateways: One Internet Gateway per cluster.

  • NAT gateways: One NAT Gateway per public subnet.

Security groups

AWS security groups provide security at the protocol and port access level; they are associated with EC2 instances and Elastic Load Balancers. Each security group contains a set of rules that filter traffic coming in and out of an EC2 instance. You must ensure the ports required for the OpenShift installation are open on your network and configured to allow access between hosts.

Group Type IP Protocol Port range

MasterSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

tcp

6443

tcp

22623

WorkerSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

BootstrapSecurityGroup

AWS::EC2::SecurityGroup

tcp

22

tcp

19531

If you are using a firewall and want to create your cluster using AWS PrivateLink, you must configure your firewall to grant access to the sites that Red Hat OpenShift Service on AWS requires.

Procedure
  1. Allowlist the following URLs that are used to install and download packages and tools:

    URL Function Port

    registry.redhat.io

    Required. Provides core components such as dev tools and operator based add-ons, and Red Hat provided container images including middleware and Universal Base Image.

    443, 80

    quay.io

    Required. Used by the cluster to download the platform container images.

    443, 80

    .quay.io

    Required. Used by the cluster to download the platform container images.

    443, 80

    sso.redhat.com

    Required. The https://cloud.redhat.com/openshift site uses authentication from sso.redhat.com to download the pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, chargeback reporting, and so on.

    443, 80

    pull.q1w2.quay.rhcloud.com

    Recommended. Provides a fallback registry used by the cluster when quay.io is not available.

    443, 80

    .q1w2.quay.rhcloud.com

    Recommended. Enables a CNAME to resolve to a regionalised endpoint.

    443, 80

    openshift.org

    Required. Provides Red Hat Enterprise Linux CoreOS (RHCOS) images.

    443, 80

    console.redhat.com

    Required. Allows interactions between the cluster and OpenShift Console Manager (OCM) to enable functionality, such as scheduling upgrades.

    443, 80

    quay-registry.s3.amazonaws.com

    Required. Used to access Quay image content in AWS.

    443, 80

    When you add a site such as quay.io to your allowlist, do not add a wildcard entry such as *.quay.io to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, then image downloads are denied when the initial download request is redirected to a host name such as cdn01.quay.io.

    CDN host names, such as cdn01.quay.io, are covered when you add a wildcard entry, such as .quay.io, in your allowlist.

    Allowlist any site that provides resources for a language or framework that your builds require.


  2. Managed clusters require telemetry to be enabled to allow Red Hat to react more quickly to problems, to better support our customers, and to better understand how product upgrades impact clusters. See About remote health monitoring for more information about how remote health monitoring data is used by Red Hat.

    URL Function Port

    https://cloud.redhat.com

    Required. Used by the cluster for the insights operator that integrates with the SaaS Red Hat Insights.

    443, 80

    cert-api.access.redhat.com

    Required. Used by telemetry.

    443, 80

    api.access.redhat.com

    Required. Used by telemetry.

    443, 80

    infogw.api.openshift.com

    Required. Used by telemetry.

    443, 80

  3. For Amazon Web Services (AWS), you must grant access to the URLs that provide the AWS API and DNS services:

    • You can grant access by allowing the .amazonaws.com wildcard:

      URL Function Requirement

      .amazonaws.com

      Required. Used to access AWS services and resources.

      443, 80

      oso-rhc4tp-docker-registry.s3-us-west-2.amazonaws.com

      Required. Used to access AWS services and resources when using strict security requirements. Review the AWS Service Endpoints in the AWS documentation to determine the exact endpoints to allow for the regions that you use.

      443, 80

    • Alternatively, you can grant access by allowing the following regional AWS service endpoints:

      URL Function Port

      ec2.<aws_region>.amazonaws.com

      Required. Used for regional access to Amazon EC2 services. EC2 instances are required to deploy the control plane and data plane functions of ROSA. Replace <aws_region> with an AWS region code, for example us-east-1.

      443, 80

      elasticloadbalancing.<aws_region>.amazonaws.com

      Required. Used to access Amazon Elastic Load Balancers (ELBs) for API and application load balancing.

      443, 80

      <cluster_id>-<shard>.<aws_region>.amazonaws.com

      Required. Used for access to the registry.

      443, 80

      The cluster ID and shard information is generated at installation time. Use rosa describe cluster --cluster=<cluster_name> to get the <cluster_id>, <shard>, and <aws_region> values.

      You can grant access to .<aws_region>.amazonaws.com when you create a cluster and later refine the firewall configuration by specifying the cluster ID and shard.

      Review the AWS Service Endpoints in the AWS documentation to determine the exact endpoints to allow for the regions that you use.

  4. Allowlist the following URLs:

    URL Function Port

    mirror.openshift.com

    Used to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator needs only a single functioning source. Required if you do not allow storage.googleapis.com/openshift-release.

    443, 80

    storage.googleapis.com/openshift-release

    Recommended. Alternative site to mirror.openshift.com/. Used to download platform release signatures that are used by the cluster to know what images to pull from quay.io.

    443, 80

    .apps.<cluster_name>.<base_domain>

    Required. Used to access the default cluster routes unless you set an ingress wildcard during installation.

    443, 80

    quay-registry.s3.amazonaws.com

    Required. Used to access Quay image content in AWS.

    443, 80

    api.openshift.com

    Required. Used to check if updates are available for the cluster.

    443, 80

    art-rhcos-ci.s3.amazonaws.com

    Required. Specifies the {op-system-first} images to download.

    443, 80

    cloud.redhat.com/openshift

    Required. Used for cluster tokens.

    443, 80

    registry.access.redhat.com

    Required. Used to access the odo CLI tool that helps developers build on OpenShift and Kubernetes.

    443, 80

    quayio-production-s3.s3.amazonaws.com

    Required. Used to install and manage clusters in an AWS environment.

    443, 80

    cm-quay-production-s3.s3.amazonaws.com

    Required. Used to install and manage clusters in an AWS environment.

    443, 80

    ec2.amazonaws.com

    Required. Used to install and manage clusters in an AWS environment.

    443, 80

    events.amazonaws.com

    Required. Used to install and manage clusters in an AWS environment.

    443, 80

    iam.amazonaws.com

    Required. Used to install and manage clusters in an AWS environment.

    443, 80

    route53.amazonaws.com

    Required. Used to install and manage clusters in an AWS environment.

    443, 80

    sts.amazonaws.com

    Required. Used to install and manage clusters in an AWS environment.

    443, 80

    ec2.<aws_region>.amazonaws.com

    Required. Region dependent. Must be added per cluster and per region.

    443, 80

    CLUSTER-NAME-k5bxz-image-registry-<aws_region>-lsiflffxtmfyikx.s3.dualstack.us-east-1.amazonaws.com

    Required. Region dependent. Must be added per cluster and per region.

    443, 80

    elasticloadbalancing.<aws_region>.amazonaws.com

    Required. Region dependent. Must be added per cluster and per region.

    443, 80

    Region is created during installation. To find the region, run:

     $ rosa describe cluster --cluster=<cluster_name>

    To retrieve the endpoint, run:

    $ oc -n openshift-image-registry get pod -l docker-registry=default -o json | jq '.items[].spec.containers[].env[] | select(.name=="REGISTRY_STORAGE_S3_BUCKET")'
  5. Operators require route access to perform health checks. Specifically, the authentication and web console Operators connect to two routes to verify that the routes work. If you are the cluster administrator and do not want to allow *.apps.<cluster_name>.<base_domain>, then you must allow these routes:

    URL Function Port

    oauth-openshift.apps.<cluster_name>.<shard>.<base_domain>

    Required.

    443

    console-openshift-console.apps.<cluster_name>.<shard>.<base_domain>, or the host name that is specified in the spec.route.hostname field of the consoles.operator/cluster object if the field is not empty

    Required.

    443

    canary-openshift-ingress-canary.apps.<cluster_name>.<shard>.s1.devshift.org

    Required.

    443

  6. If you use a default Red Hat Network Time Protocol (NTP) server, allowlist the following URLs:

    • 1.rhel.pool.ntp.org

    • 2.rhel.pool.ntp.org

    • 3.rhel.pool.ntp.org

      If you do not use a default Red Hat NTP server, verify the NTP server for your platform and allowlist it in your firewall.

  7. Allowlist the following OpenShift Dedicated URLs:

    URL Function Port

    api.pagerduty.com and events.pagerduty.com

    Required. This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on.

    443

    api.deadmanssnitch.com and nosnch.in

    Required. Alerting service used by OpenShift Dedicated to send periodic pings that indicate whether the cluster is available and running.

    443

    sftp.access.redhat.com

    Recommended. The FTP server used by must-gather-operator to upload diagnostic logs to help troubleshoot issues with the cluster.

    443

    .osdsecuritylogs.splunkcloud.com inputs1.osdsecuritylogs.splunkcloud.com inputs2.osdsecuritylogs.splunkcloud.com inputs4.osdsecuritylogs.splunkcloud.com inputs5.osdsecuritylogs.splunkcloud.com inputs6.osdsecuritylogs.splunkcloud.com inputs7.osdsecuritylogs.splunkcloud.com inputs8.osdsecuritylogs.splunkcloud.com inputs9.osdsecuritylogs.splunkcloud.com inputs10.osdsecuritylogs.splunkcloud.com inputs11.osdsecuritylogs.splunkcloud.com inputs12.osdsecuritylogs.splunkcloud.com inputs13.osdsecuritylogs.splunkcloud.com inputs14.osdsecuritylogs.splunkcloud.com inputs15.osdsecuritylogs.splunkcloud.com

    http-inputs-osdsecuritylogs.splunkcloud.com

    Required. Used by the splunk-forwarder-operator as a logging forwarding endpoint to be used by Red Hat SRE for log-based alerting.

    443

    observatorium.api.openshift.com

    Required. Used for Managed OpenShift-specific telemetry.

    443

  8. Allowlist any site that provides resources for a language or framework that your builds require.

  9. Allowlist any outbound URLs that depend on the languages and frameworks used in OpenShift. See OpenShift Outbound URLs to Allow for a list of recommended URLs to be allowed on the firewall or proxy.

Additional resources