×

Red Hat OpenShift Service on AWS (ROSA) provides a model that allows Red Hat to deploy clusters into a customer’s existing Amazon Web Service (AWS) account.

You must ensure that the prerequisites are met before installing ROSA. This requirements document does not apply to AWS Security Token Service (STS). If you are using STS, see the STS-specific requirements.

AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS (ROSA) because it provides enhanced security.

Customer Requirements

Red Hat OpenShift Service on AWS (ROSA) clusters must meet several prerequisites before they can be deployed.

In order to create the cluster, the user must be logged in as an IAM user and not an assumed role or STS user.

Account

  • The customer ensures that the AWS limits are sufficient to support Red Hat OpenShift Service on AWS provisioned within the customer’s AWS account.

  • The customer’s AWS account should be in the customer’s AWS Organizations with the applicable service control policy (SCP) applied.

    It is not a requirement that the customer’s account be within the AWS Organizations or for the SCP to be applied, however Red Hat must be able to perform all the actions listed in the SCP without restriction.

  • The customer’s AWS account should not be transferable to Red Hat.

  • The customer may not impose AWS usage restrictions on Red Hat activities. Imposing restrictions will severely hinder Red Hat’s ability to respond to incidents.

  • The customer may deploy native AWS services within the same AWS account.

    Customers are encouraged, but not mandated, to deploy resources in a Virtual Private Cloud (VPC) separate from the VPC hosting Red Hat OpenShift Service on AWS and other Red Hat supported services.

Access requirements

  • To appropriately manage the Red Hat OpenShift Service on AWS service, Red Hat must have the AdministratorAccess policy applied to the administrator role at all times. This requirement does not apply if you are using AWS Security Token Service (STS).

    This policy only provides Red Hat with permissions and capabilities to change resources in the customer-provided AWS account.

  • Red Hat must have AWS console access to the customer-provided AWS account. This access is protected and managed by Red Hat.

  • The customer must not utilize the AWS account to elevate their permissions within the Red Hat OpenShift Service on AWS cluster.

  • Actions available in the Red Hat OpenShift Service on AWS (ROSA) CLI, rosa, or OpenShift Cluster Manager console must not be directly performed in the customer’s AWS account.

Support requirements

  • Red Hat recommends that the customer have at least Business Support from AWS.

  • Red Hat has authority from the customer to request AWS support on their behalf.

  • Red Hat has authority from the customer to request AWS resource limit increases on the customer’s account.

  • Red Hat manages the restrictions, limitations, expectations, and defaults for all Red Hat OpenShift Service on AWS clusters in the same manner, unless otherwise specified in this requirements section.

Security requirements

  • Volume snapshots will remain within the customer’s AWS account and customer-specified region.

  • Red Hat must have ingress access to EC2 hosts and the API server from allow-listed IP addresses.

  • Red Hat must have egress allowed to forward system and audit logs to a Red Hat managed central logging stack.

Required customer procedure

Complete these steps before deploying Red Hat OpenShift Service on AWS (ROSA).

Procedure
  1. If you, as the customer, are utilizing AWS Organizations, then you must use an AWS account within your organization or create a new one.

  2. To ensure that Red Hat can perform necessary actions, you must either create a service control policy (SCP) or ensure that none is applied to the AWS account.

  3. Attach the SCP to the AWS account.

  4. Follow the ROSA procedures for setting up the environment.

Minimum set of effective permissions for service control policies (SCP)

Service control policies (SCP) are a type of organization policy that manages permissions within your organization. SCPs ensure that accounts within your organization stay within your defined access control guidelines. These polices are maintained in AWS Organizations and control the services that are available within the attached AWS accounts. SCP management is the responsibility of the customer.

The minimum SCP requirement does not apply when using AWS Security Token Service (STS). For more information about STS, see AWS prerequisites for ROSA with STS.

Verify that your service control policy (SCP) does not restrict any of these required permissions.

Service Actions Effect

Required

Amazon EC2

All

Allow

Amazon EC2 Auto Scaling

All

Allow

Amazon S3

All

Allow

Identity And Access Management

All

Allow

Elastic Load Balancing

All

Allow

Elastic Load Balancing V2

All

Allow

Amazon CloudWatch

All

Allow

Amazon CloudWatch Events

All

Allow

Amazon CloudWatch Logs

All

Allow

AWS EC2 Instance Connect

SendSerialConsoleSSHPublicKey

Allow

AWS Support

All

Allow

AWS Key Management Service

All

Allow

AWS Security Token Service

All

Allow

AWS Tiro

CreateQuery

GetQueryAnswer

GetQueryExplanation

Allow

AWS Marketplace

Subscribe

Unsubscribe

View Subscriptions

Allow

AWS Resource Tagging

All

Allow

AWS Route53 DNS

All

Allow

AWS Service Quotas

ListServices

GetRequestedServiceQuotaChange

GetServiceQuota

RequestServiceQuotaIncrease

ListServiceQuotas

Allow

Optional

AWS Billing

ViewAccount

Viewbilling

ViewUsage

Allow

AWS Cost and Usage Report

All

Allow

AWS Cost Explorer Services

All

Allow

Red Hat managed IAM references for AWS

Red Hat is responsible for creating and managing the following Amazon Web Services (AWS) resources: IAM policies, IAM users, and IAM roles.

IAM Policies

IAM policies are subject to modification as the capabilities of Red Hat OpenShift Service on AWS change.

  • The AdministratorAccess policy is used by the administration role. This policy provides Red Hat the access necessary to administer the Red Hat OpenShift Service on AWS (ROSA) cluster in the customer’s AWS account.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "*",
                "Resource": "*",
                "Effect": "Allow"
            }
        ]
    }

IAM users

The osdManagedAdmin user is created immediately after installing ROSA into the customer’s AWS account.

Provisioned AWS Infrastructure

This is an overview of the provisioned Amazon Web Services (AWS) components on a deployed Red Hat OpenShift Service on AWS (ROSA) cluster. For a more detailed listing of all provisioned AWS components, see the OpenShift Container Platform documentation.

EC2 instances

AWS EC2 instances are required for deploying the control plane and data plane functions of ROSA in the AWS public cloud.

Instance types can vary for control plane and infrastructure nodes, depending on the worker node count. At a minimum, the following EC2 instances will be deployed:

  • Three m5.2xlarge control plane nodes

  • Two r5.xlarge infrastructure nodes

  • Two m5.xlarge customizable worker nodes

For further guidance on worker node counts, see the information about initial planning considerations in the "Limits and scalability" topic listed in the "Additional resources" section of this page.

Amazon Elastic Block Store storage

Amazon Elastic Block Store (Amazon EBS) block storage is used for both local node storage and persistent volume storage.

Volume requirements for each EC2 instance:

  • Control Plane Volume

    • Size: 350GB

    • Type: gp3

    • Input/Output Operations Per Second: 1000

  • Infrastructure Volume

    • Size: 300GB

    • Type: gp3

    • Input/Output Operations Per Second: 900

  • Worker Volume

    • Size: 300GB

    • Type: gp3

    • Input/Output Operations Per Second: 900

Clusters deployed before the release of OpenShift Container Platform 4.11 use gp2 type storage by default.

Elastic Load Balancing

Up to two Network Load Balancers for API and up to two Classic Load Balancers for application router. For more information, see the ELB documentation for AWS.

S3 storage

The image registry is backed by AWS S3 storage. Pruning of resources is performed regularly to optimize S3 usage and cluster performance.

Two buckets are required with a typical size of 2TB each.

VPC

Customers should expect to see one VPC per cluster. Additionally, the VPC will need the following configurations:

  • Subnets: Two subnets for a cluster with a single availability zone, or six subnets for a cluster with multiple availability zones.

    A public subnet connects directly to the internet through an internet gateway. A private subnet connects to the internet through a network address translation (NAT) gateway.

  • Route tables: One route table per private subnet, and one additional table per cluster.

  • Internet gateways: One Internet Gateway per cluster.

  • NAT gateways: One NAT Gateway per public subnet.

VPC Reference Architecture
Figure 1. Sample VPC Architecture

Security groups

AWS security groups provide security at the protocol and port access level; they are associated with EC2 instances and Elastic Load Balancing (ELB) load balancers. Each security group contains a set of rules that filter traffic coming in and out of one or more EC2 instances. You must ensure the ports required for the OpenShift installation are open on your network and configured to allow access between hosts.

Table 1. Required ports for default security groups
Group Type IP Protocol Port range

MasterSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

tcp

6443

tcp

22623

WorkerSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

BootstrapSecurityGroup

AWS::EC2::SecurityGroup

tcp

22

tcp

19531

Additional custom security groups

When you create a cluster using an existing non-managed VPC, you can add additional custom security groups during cluster creation. Custom security groups are subject to the following limitations:

  • You must create the custom security groups in AWS before you create the cluster. For more information, see Amazon EC2 security groups for Linux instances.

  • You must associate the custom security groups with the VPC that the cluster will be installed into. Your custom security groups cannot be associated with another VPC.

  • You might need to request additional quota for your VPC if you are adding additional custom security groups. For information on AWS quota requirements for ROSA, see Required AWS service quotas in Prepare your environment. For information on requesting an AWS quota increase, see Requesting a quota increase.

Only ROSA clusters deployed with PrivateLink can use a firewall to control egress traffic.

This section provides the necessary details that enable you to control egress traffic from your Red Hat OpenShift Service on AWS cluster. If you are using a firewall to control egress traffic, you must configure your firewall to grant access to the domain and port combinations below. Red Hat OpenShift Service on AWS requires this access to provide a fully managed OpenShift service.

Procedure
  1. Allowlist the following URLs that are used to install and download packages and tools:

    Domain Port Function

    registry.redhat.io

    443

    Provides core container images.

    quay.io

    443

    Provides core container images.

    .quay.io

    443

    Provides core container images.

    sso.redhat.com

    443

    Required. The https://console.redhat.com/openshift site uses authentication from sso.redhat.com to download the pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, chargeback reporting, and so on.

    quay-registry.s3.amazonaws.com

    443

    Provides core container images.

    ocm-quay-production-s3.s3.amazonaws.com

    443

    Provides core container images.

    quayio-production-s3.s3.amazonaws.com

    443

    Provides core container images.

    cart-rhcos-ci.s3.amazonaws.com

    443

    Provides Red Hat Enterprise Linux CoreOS (RHCOS) images.

    openshift.org

    443

    Provides Red Hat Enterprise Linux CoreOS (RHCOS) images.

    registry.access.redhat.com [1]

    443

    Hosts all the container images that are stored on the Red Hat Ecosytem Catalog. Additionally, the registry provides access to the odo CLI tool that helps developers build on OpenShift and Kubernetes.

    registry.connect.redhat.com

    443

    Required for all third-party images and certified Operators.

    console.redhat.com

    443

    Required. Allows interactions between the cluster and OpenShift Console Manager to enable functionality, such as scheduling upgrades.

    sso.redhat.com

    443

    The https://console.redhat.com/openshift site uses authentication from sso.redhat.com.

    pull.q1w2.quay.rhcloud.com

    443

    Provides core container images as a fallback when quay.io is not available.

    .q1w2.quay.rhcloud.com

    443

    Provides core container images as a fallback when quay.io is not available.

    www.okd.io

    443

    The openshift.org site redirects through www.okd.io.

    www.redhat.com

    443

    The sso.redhat.com site redirects through www.redhat.com.

    aws.amazon.com

    443

    The iam.amazonaws.com and sts.amazonaws.com sites redirect through aws.amazon.com.

    catalog.redhat.com

    443

    The registry.access.redhat.com and https://registry.redhat.io sites redirect through catalog.redhat.com.

    dvbwgdztaeq9o.cloudfront.net [2]

    443

    Used by ROSA for STS implementation with managed OIDC configuration.

    1. In a firewall environment, ensure that the access.redhat.com resource is on the allowlist. This resource hosts a signature store that a container client requires for verifying images when pulling them from registry.access.redhat.com.

    2. The string of alphanumeric characters before cloudfront.net could change if there is a major cloudfront outage that requires redirecting the resource.

    When you add a site such as quay.io to your allowlist, do not add a wildcard entry such as .quay.io to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, then image downloads are denied when the initial download request is redirected to a host name such as cdn01.quay.io.

    CDN host names, such as cdn01.quay.io, are covered when you add a wildcard entry, such as .quay.io, in your allowlist.

  2. Allowlist the following telemetry URLs:

    Domain Port Function

    cert-api.access.redhat.com

    443

    Required for telemetry.

    api.access.redhat.com

    443

    Required for telemetry.

    infogw.api.openshift.com

    443

    Required for telemetry.

    console.redhat.com

    443

    Required for telemetry and Red Hat Insights.

    cloud.redhat.com/api/ingress

    443

    Required for telemetry and Red Hat Insights.

    observatorium-mst.api.openshift.com

    443

    Required for managed OpenShift-specific telemetry.

    observatorium.api.openshift.com

    443

    Required for managed OpenShift-specific telemetry.

    Managed clusters require enabling telemetry to allow Red Hat to react more quickly to problems, better support the customers, and better understand how product upgrades impact clusters. For more information about how remote health monitoring data is used by Red Hat, see About remote health monitoring in the Additional resources section.

  3. Allowlist the following Amazon Web Services (AWS) API URls:

    Domain Port Function

    .amazonaws.com

    443

    Required to access AWS services and resources.

    Alternatively, if you choose to not use a wildcard for Amazon Web Services (AWS) APIs, you must allowlist the following URLs:

    Domain Port Function

    ec2.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment.

    events.<aws_region>.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment.

    iam.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment.

    route53.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment.

    sts.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment, for clusters configured to use the global endpoint for AWS STS.

    sts.<aws_region>.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment, for clusters configured to use regionalized endpoints for AWS STS. See AWS STS regionalized endpoints for more information.

    tagging.us-east-1.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment. This endpoint is always us-east-1, regardless of the region the cluster is deployed in.

    ec2.<aws_region>.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment.

    elasticloadbalancing.<aws_region>.amazonaws.com

    443

    Used to install and manage clusters in an AWS environment.

    servicequotas.<aws_region>.amazonaws.com

    443

    Required. Used to confirm quotas for deploying the service.

    tagging.<aws_region>.amazonaws.com

    443

    Allows the assignment of metadata about AWS resources in the form of tags.

  4. Allowlist the following OpenShift URLs:

    Domain Port Function

    mirror.openshift.com

    443

    Used to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator (CVO) needs only a single functioning source.

    storage.googleapis.com/openshift-release (Recommended)

    443

    Alternative site to mirror.openshift.com/. Used to download platform release signatures that are used by the cluster to know what images to pull from quay.io.

    api.openshift.com

    443

    Used to check if updates are available for the cluster.

  5. Allowlist the following site reliability engineering (SRE) and management URLs:

    Domain Port Function

    api.pagerduty.com

    443

    This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on.

    events.pagerduty.com

    443

    This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on.

    api.deadmanssnitch.com

    443

    Alerting service used by Red Hat OpenShift Service on AWS to send periodic pings that indicate whether the cluster is available and running.

    nosnch.in

    443

    Alerting service used by Red Hat OpenShift Service on AWS to send periodic pings that indicate whether the cluster is available and running.

    .osdsecuritylogs.splunkcloud.com OR inputs1.osdsecuritylogs.splunkcloud.com inputs2.osdsecuritylogs.splunkcloud.com inputs4.osdsecuritylogs.splunkcloud.com inputs5.osdsecuritylogs.splunkcloud.com inputs6.osdsecuritylogs.splunkcloud.com inputs7.osdsecuritylogs.splunkcloud.com inputs8.osdsecuritylogs.splunkcloud.com inputs9.osdsecuritylogs.splunkcloud.com inputs10.osdsecuritylogs.splunkcloud.com inputs11.osdsecuritylogs.splunkcloud.com inputs12.osdsecuritylogs.splunkcloud.com inputs13.osdsecuritylogs.splunkcloud.com inputs14.osdsecuritylogs.splunkcloud.com inputs15.osdsecuritylogs.splunkcloud.com

    9997

    Used by the splunk-forwarder-operator as a logging forwarding endpoint to be used by Red Hat SRE for log-based alerting.

    http-inputs-osdsecuritylogs.splunkcloud.com

    443

    Required. Used by the splunk-forwarder-operator as a logging forwarding endpoint to be used by Red Hat SRE for log-based alerting.

    sftp.access.redhat.com (Recommended)

    22

    The SFTP server used by must-gather-operator to upload diagnostic logs to help troubleshoot issues with the cluster.

  6. Allowlist the following URLs for optional third-party content:

    Domain Port Function

    registry.connect.redhat.com

    443

    Required for all third-party-images and certified operators.

    rhc4tp-prod-z8cxf-image-registry-us-east-1-evenkyleffocxqvofrk.s3.dualstack.us-east-1.amazonaws.com

    443

    Provides access to container images hosted on registry.connect.redhat.com

    oso-rhc4tp-docker-registry.s3-us-west-2.amazonaws.com

    443

    Required for Sonatype Nexus, F5 Big IP operators.

  7. If you did not allow a wildcard for Amazon Web Services (AWS) APIs, you must also allow the S3 bucket used for the internal OpenShift registry. To retrieve that endpoint, run the following command after the cluster is successfully provisioned:

    $ oc -n openshift-image-registry get pod -l docker-registry=default -o json | jq '.items[].spec.containers[].env[] | select(.name=="REGISTRY_STORAGE_S3_BUCKET")'

    The S3 endpoint should be in the following format:

    '<cluster-name>-<random-string>-image-registry-<cluster-region>-<random-string>.s3.dualstack.<cluster-region>.amazonaws.com'.
  8. Allowlist any site that provides resources for a language or framework that your builds require.

  9. Allowlist any outbound URLs that depend on the languages and frameworks used in OpenShift. See OpenShift Outbound URLs to Allow for a list of recommended URLs to be allowed on the firewall or proxy.