Prerequisites for Red Hat OpenShift Service on AWS

Red Hat OpenShift Service on AWS (ROSA) provides a model that allows Red Hat to deploy clusters into a customer’s existing Amazon Web Service (AWS) account.

You must ensure that the prerequisites are met before installing ROSA.

Deployment Prerequisites

To deploy Red Hat OpenShift Service on AWS (ROSA) into your existing Amazon Web Services (AWS) account, Red Hat requires that several prerequisites are met.

Red Hat recommends the usage of AWS Organizations to manage multiple AWS accounts. The AWS Organizations, managed by the customer, host multiple AWS accounts. There is a root account in the organization that all accounts will refer to in the account hierarchy.

It is a best practice for the ROSA cluster to be hosted in an AWS account within an AWS Organizational Unit. A Service Control Policy (SCP) is created and applied to the AWS Organizational Unit that manages what services the AWS sub-accounts are permitted to access. The SCP applies only to available permissions within a single AWS account for all AWS sub-accounts within the Organizational Unit. It is also possible to apply a SCP to a single AWS account. All other accounts in the customer’s AWS Organizations are managed in whatever manner the customer requires. Red Hat Site Reliability Engineers (SRE) will not have any control over SCPs within AWS Organizations.

Customer Requirements

Red Hat OpenShift Service on AWS (ROSA) clusters must meet several prerequisites before they can be deployed.

In order to create the cluster, the user must be logged in as an IAM user and not an assumed role or STS user.

Account

  • The customer ensures that the AWS limits are sufficient to support Red Hat OpenShift Service on AWS provisioned within the customer’s AWS account.

  • The customer’s AWS account should be in the customer’s AWS Organizations with the applicable Service Control Policy (SCP) applied.

    It is not a requirement that the customer’s account be within the AWS Organizations or for the SCP to be applied, however Red Hat must be able to perform all the actions listed in the SCP without restriction.

  • The customer’s AWS account should not be transferable to Red Hat.

  • The customer may not impose AWS usage restrictions on Red Hat activities. Imposing restrictions will severely hinder Red Hat’s ability to respond to incidents.

  • The customer may deploy native AWS services within the same AWS account.

    Customers are encouraged, but not mandated, to deploy resources in a Virtual Private Cloud (VPC) separate from the VPC hosting Red Hat OpenShift Service on AWS and other Red Hat supported services.

Access requirements

  • To appropriately manage the Red Hat OpenShift Service on AWS service, Red Hat must have the AdministratorAccess policy applied to the administrator role at all times.

    This policy only provides Red Hat with permissions and capabilities to change resources in the customer-provided AWS account.

  • Red Hat must have AWS console access to the customer-provided AWS account. This access is protected and managed by Red Hat.

  • The customer must not utilize the AWS account to elevate their permissions within the Red Hat OpenShift Service on AWS cluster.

  • Actions available in the rosa CLI utility or OpenShift Cluster Manager (OCM) console must not be directly performed in the customer’s AWS account.

Support requirements

  • Red Hat recommends that the customer have at least Business Support from AWS.

  • Red Hat has authority from the customer to request AWS support on their behalf.

  • Red Hat has authority from the customer to request AWS resource limit increases on the customer’s account.

  • Red Hat manages the restrictions, limitations, expectations, and defaults for all Red Hat OpenShift Service on AWS clusters in the same manner, unless otherwise specified in this requirements section.

Security requirements

  • Volume snapshots will remain within the customer’s AWS account and customer-specified region.

  • Red Hat must have ingress access to EC2 hosts and the API server from allow-listed IP addresses.

  • Red Hat must have egress allowed to forward system and audit logs to a Red Hat managed central logging stack.

Required customer procedure

Complete these steps before deploying Red Hat OpenShift Service on AWS (ROSA).

Procedure
  1. If you, as the customer, are utilizing AWS Organizations, then you must use an AWS account within your organization or create a new one.

  2. To ensure that Red Hat can perform necessary actions, you must either create a Service Control Policy (SCP) or ensure that none is applied to the AWS account.

  3. Attach the SCP to the AWS account.

  4. Follow the ROSA procedures for setting up the environment.

Minimum required Service Control Policy (SCP)

Service Control Policy (SCP) management is the responsibility of the customer. These policies are maintained in the AWS Organizations and control what services are available within the attached AWS accounts.

Service Actions Effect

Required

Amazon EC2

All

Allow

Amazon EC2 Auto Scaling

All

Allow

Amazon S3

All

Allow

Identity And Access Management

All

Allow

Elastic Load Balancing

All

Allow

Elastic Load Balancing V2

All

Allow

Amazon CloudWatch

All

Allow

Amazon CloudWatch Events

All

Allow

Amazon CloudWatch Logs

All

Allow

AWS Support

All

Allow

AWS Key Management Service

All

Allow

AWS Security Token Service

All

Allow

AWS Resource Tagging

All

Allow

AWS Route53 DNS

All

Allow

AWS Service Quotas

ListServices

GetRequestedServiceQuotaChange

GetServiceQuota

RequestServiceQuotaIncrease

ListServiceQuotas

Allow

Optional

AWS Billing

ViewAccount

Viewbilling

ViewUsage

Allow

AWS Cost and Usage Report

All

Allow

AWS Cost Explorer Services

All

Allow

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "events:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "support:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sts:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "tag:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "servicequotas:ListServices",
                "servicequotas:GetRequestedServiceQuotaChange",
                "servicequotas:GetServiceQuota",
                "servicequotas:RequestServiceQuotaIncrease",
                "servicequotas:ListServiceQuotas"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Red Hat managed IAM references for AWS

Red Hat is responsible for creating and managing the following Amazon Web Services (AWS) resources: IAM policies, IAM users, and IAM roles.

IAM Policies

IAM policies are subject to modification as the capabilities of Red Hat OpenShift Service on AWS change.

  • The AdministratorAccess policy is used by the administration role. This policy provides Red Hat the access necessary to administer the Red Hat OpenShift Service on AWS (ROSA) cluster in the customer’s AWS account.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": "*",
                "Resource": "*",
                "Effect": "Allow"
            }
        ]
    }

IAM users

The osdManagedAdmin user is created immediately after installing ROSA into the customer’s AWS account.

Provisioned AWS Infrastructure

This is an overview of the provisioned Amazon Web Services (AWS) components on a deployed Red Hat OpenShift Service on AWS (ROSA) cluster. For a more detailed listing of all provisioned AWS components, see the OpenShift Container Platform documentation.

EC2 instances

AWS EC2 instances are required for deploying the control plane and data plane functions of ROSA in the AWS public cloud.

Instance types can vary for control plane and infrastructure nodes, depending on the worker node count.

  • Three m5.xlarge minimum (control plane nodes)

  • Two r5.xlarge minimum (infrastructure nodes)

  • Two m5.xlarge minimum but highly variable (worker nodes)

Elastic Block Storage storage

Amazon EBS block storage is used for both local node storage and persistent volume storage.

Volume requirements for each EC2 instance:

  • Control Plane Volume

    • Size: 350GB

    • Type: io1

    • Input/Output Operations Per Second: 1000

  • Infrastructure Volume

    • Size: 300GB

    • Type: gp2

    • Input/Output Operations Per Second: 100

  • Worker Volume

    • Size: 300GB

    • Type: gp2

    • Input/Output Operations Per Second: 100

Elastic load balancers

Up to two Network Elastic Load Balancers (ELBs) for API and up to two Classic ELBs for application router. For more information, see the ELB documentation for AWS.

S3 storage

The image registry and Elastic Block Store (EBS) volume snapshots are backed by AWS S3 storage. Pruning of resources is performed regularly to optimize S3 usage and cluster performance.

Two buckets are required with a typical size of 2TB each.

VPC

Customers should expect to see one VPC per cluster. Additionally, the VPC will need the following configurations:

  • Subnets: Two subnets for a cluster with a single availability zone, or six subnets for a cluster with multiple availability zones.

  • Router tables: One router table per private subnet, and one additional table per cluster.

  • Internet gateways: One Internet Gateway per cluster.

  • NAT gateways: One NAT Gateway per public subnet.

Security groups

AWS security groups provide security at the protocol and port access level; they are associated with EC2 instances and Elastic Load Balancers. Each security group contains a set of rules that filter traffic coming in and out of an EC2 instance. You must ensure the ports required for the OpenShift installation are open on your network and configured to allow access between hosts.

Group Type IP Protocol Port range

MasterSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

tcp

6443

tcp

22623

WorkerSecurityGroup

AWS::EC2::SecurityGroup

icmp

0

tcp

22

BootstrapSecurityGroup

AWS::EC2::SecurityGroup

tcp

22

tcp

19531

If you are using a firewall and want to create your cluster using AWS PrivateLink, you must configure your firewall to grant access to the sites that Red Hat OpenShift Service on AWS requires.

See Configuring your firewall for OpenShift Container Platform for information about granting access to sites that OCP requires.

Procedure
  1. Allowlist the following URLs that are used to install and download packages and tools:

    URL Function Requirement

    registry.redhat.io

    Provides core components such as dev tools and operator based add-ons, and Red Hat provided container images including middleware and Universal Base Image.

    Required

    quay.io

    Used by the cluster to download the platform container images

    Required

    sso.redhat.com

    The https://cloud.redhat.com/openshift site uses authentication from sso.redhat.com to download the pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, chargeback reporting, and so on.

    Required

    pull.q1w2.quay.rhcloud.com

    Provides a fallback registry used by the cluster when quay.io is not available.

    Recommended

    When you add a site such as quay.io to your allowlist, do not add a wildcard entry such as *.quay.io to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, then image downloads are denied when the initial download request is redirected to a host name such as cdn01.quay.io.

  2. Managed clusters require Telemetry to be enabled to allow Red Hat to react more quickly to problems and better support our customers, as well as better understand how product upgrades impact clusters. See About remote health monitoring for more information about how remote health monitoring data is used by Red Hat.

    URL Function Requirement

    https://cloud.redhat.com

    Used by the cluster for the insights operator that integrates with the SaaS Red Hat Insights

    Required

  3. For Amazon Web Services (AWS), you must grant access to the URLs that provide the AWS API and DNS:

    URL Function Requirement

    *.amazonaws.com

    Required to access AWS services and resources. Review the AWS Service Endpoints in the AWS documentation to determine the exact endpoints to allow for the regions that you use.

    Required

  4. Allowlist the following URLs:

    URL Function Requirement

    mirror.openshift.com

    Required to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator needs only a single functioning source.

    Required

    storage.googleapis.com/openshift-release

    Alternative site to download platform release signatures that are used by the cluster to know what images to pull from quay.io.

    Recommended

    quay-registry.s3.amazonaws.com

    Used to access Quay image content in AWS.

    Required

    api.openshift.com

    Required to check if updates are available for the cluster.

    Required

    art-rhcos-ci.s3.amazonaws.com

    Specifies the {op-system-first} images to download.

    Required

    cloud.redhat.com/openshift

    Required for your cluster token.

    Required

    registry.access.redhat.com

    Access to the odo CLI tool that helps developers build on OpenShift and Kubernetes.

    Required

  5. Allowlist the following OpenShift Dedicated URLs:

    URL Function Requirement

    api.pagerduty.com and events.pagerduty.com

    This alerting service is used by the in-cluster alertmanager to send alerts notifying SREs of an event to take action on.

    Required

    api.deadmanssnitch.com and nosnch.in

    Alerting service used by OpenShift Dedicated to send periodic pings that indicate whether the cluster is available and running.

    Required

    sftp.access.redhat.com

    FTP server used by must-gather-operator to upload diagnostic logs to help troubleshoot issues with the cluster.

    Recommended

    *.osdsecuritylogs.splunkcloud.com

    Used by the splunk-forwarder-operator as a logging forwarding endpoint to be used by the SRE Platform Team for log-based alerting.

    Required

    observatorium.api.openshift.com

    Used for Managed OpenShift-specific telemetry.

    Required

  6. Allowlist any site that provides resources for a language or framework that your builds require.

  7. Allowlist any outbound URLs that depend on the languages and frameworks used in OpenShift. See OpenShift Outbound URLs to Allow for a list of recommended URLs to be allowed on the firewall or proxy.

Additional resources