This documentation outlines the service definition for the Red Hat OpenShift Service on AWS (ROSA) managed service.

Account management

This section provides information about the service definition for Red Hat OpenShift Service on AWS account management.

Billing

Red Hat OpenShift Service on AWS is billed through Amazon Web Services (AWS) based on the usage of AWS components used by the service, such as load balancers, storage, EC2 instances, other components, and Red Hat subscriptions for the OpenShift service.

Any additional Red Hat software must be purchased separately.

Cluster self-service

Customers can self-service their clusters, including, but not limited to:

  • Create a cluster

  • Delete a cluster

  • Add or remove an identity provider

  • Add or remove a user from an elevated group

  • Configure cluster privacy

  • Add or remove machine pools and configure autoscaling

  • Define upgrade policies

These tasks can be self-serviced using the rosa CLI utility.

Compute

Single availability zone clusters require a minimum of 3 control planes, 2 infrastructure nodes, and 2 worker nodes deployed to a single availability zone.

Multiple availability zone clusters require a minimum of 3 control planes. 3 infrastructure nodes, and 3 worker nodes. Additional nodes must be purchased in multiples of three to maintain proper node distribution.

All Red Hat OpenShift Service on AWS clusters support a maximum of 180 worker nodes.

The Default machine pool node type and size cannot be changed after the cluster is created.

Control plane and infrastructure nodes are deployed and managed by Red Hat. There are at least 3 control plane nodes that handle etcd- and API-related workloads. There are at least 2 infrastructure nodes that handle metrics, routing, the web console, and other workloads. Control plane and infrastructure nodes are strictly for Red Hat workloads to operate the service, and customer workloads are not permitted to be deployed on these nodes.

1 vCPU core and 1 GiB of memory are reserved on each worker node to run processes required as part of the managed service. This includes, but is not limited to, audit log aggregation, metrics collection, DNS, image registry, and SDN.

AWS compute types

Red Hat OpenShift Service on AWS offers the following worker node types and sizes:

General purpose

  • M5.xlarge (4 vCPU, 16 GiB)

  • M5.2xlarge (8 vCPU, 32 GiB)

  • M5.4xlarge (16 vCPU, 64 GiB)

  • M5.8xlarge (32 vCPU, 128 GiB)

  • M5.12xlarge (48 vCPU, 192 GiB)

  • M5.16xlarge (64 vCPU, 256 GiB)

  • M5.24xlarge (96 vCPU, 384 GiB)

Memory-optimized

  • R5.xlarge (4 vCPU, 32 GiB)

  • R5.2xlarge (8 vCPU, 64 GiB)

  • R5.4xlarge (16 vCPU, 128 GiB)

  • R5.8xlarge (32 vCPU, 256 GiB)

  • R5.12xlarge (48 vCPU, 384 GiB)

  • R5.16xlarge (64 vCPU, 512 GiB)

  • R5.24xlarge (96 vCPU, 768 GiB)

Compute-optimized

  • C5.2xlarge (8 vCPU, 16 GiB)

  • C5.4xlarge (16 vCPU, 32 GiB)

  • C5.9xlarge (36 vCPU, 72 GiB)

  • C5.12xlarge (48 vCPU, 96 GiB)

  • C5.18xlarge (72 vCPU, 144 GiB)

  • C5.24xlarge (96 vCPU, 192 GiB)

Regions and availability zones

The following AWS regions are supported by Red Hat OpenShift 4 and are supported for Red Hat OpenShift Service on AWS. Note: China and GovCloud (US) regions are not supported, regardless of their support on OpenShift 4.

  • ap-northeast-1 (Tokyo)

  • ap-northeast-2 (Seoul)

  • ap-south-1 (Mumbai)

  • ap-southeast-1 (Singapore)

  • ap-southeast-2 (Sydney)

  • ca-central-1 (Central)

  • eu-central-1 (Frankfurt)

  • eu-north-1 (Stockholm)

  • eu-west-1 (Ireland)

  • eu-west-2 (London)

  • eu-west-3 (Paris)

  • me-south-1 (Bahrain)

  • sa-east-1 (São Paulo)

  • us-east-1 (N. Virginia)

  • us-east-2 (Ohio)

  • us-west-1 (N. California)

  • us-west-2 (Oregon)

Multiple availability zone clusters can only be deployed in regions with at least 3 availability zones. For more information, see the Regions and Availability Zones section in the AWS documentation.

Each new Red Hat OpenShift Service on AWS cluster is installed within an installer-created or preexisting Virtual Private Cloud (VPC) in a single region, with the option to deploy into a single availability zone (Single-AZ) or across multiple availability zones (Multi-AZ). This provides cluster-level network and resource isolation, and enables cloud-provider VPC settings, such as VPN connections and VPC Peering. Persistent volumes (PVs) are backed by AWS Elastic Block Storage (EBS), and are specific to the availability zone in which they are provisioned. Persistent volume claims (PVCs) do not bind to a volume until the associated pod resource is assigned into a specific availability zone to prevent unschedulable pods. Availability zone-specific resources are only usable by resources in the same availability zone.

The region and the choice of single or multiple availability zone cannot be changed after a cluster has been deployed.

Service Level Agreement (SLA)

Any SLAs for the service itself are defined in Appendix 4 of the Red Hat Enterprise Agreement Appendix 4 (Online Subscription Services).

Support

Red Hat OpenShift Service on AWS includes Red Hat Premium Support, which can be accessed by using the Red Hat Customer Portal.

See Red Hat OpenShift Service on AWS SLAs for support response times.

AWS support is subject to a customer’s existing support contract with AWS.

Logging

Red Hat OpenShift Service on AWS provides optional integrated log forwarding to AWS CloudWatch.

Cluster audit logging

Cluster audit logs are always enabled. Audit logs are streamed to a log aggregation system outside the cluster VPC for automated security analysis and secure retention for 1 year. Red Hat controls the log aggregation system. Customers do not have access. Customers can receive a copy of their cluster’s audit logs upon request through a support ticket. Audit log requests must specify a date and time range not to exceed 21 days. When requesting audit logs, customers should be aware that audit logs are many GB per day in size.

Application logging

Application logs sent to STDOUT are collected by Fluentd and forwarded to AWS CloudWatch through the cluster logging stack, if it is installed.

Monitoring

This section provides information about the service definition for Red Hat OpenShift Service on AWS monitoring.

Cluster metrics

Red Hat OpenShift Service on AWS clusters come with an integrated Prometheus stack for cluster monitoring including CPU, memory, and network-based metrics. This is accessible through the web console. These metrics also allow for horizontal pod autoscaling based on CPU or memory metrics provided by an Red Hat OpenShift Service on AWS user.

Cluster status notification

Red Hat communicates the health and status of Red Hat OpenShift Service on AWS clusters through a combination of a cluster dashboard available in the OpenShift Cluster Manager (OCM), and email notifications sent to the email address of the contact that originally deployed the cluster, and any additional contacts specified by the customer.

Networking

This section provides information about the service definition for Red Hat OpenShift Service on AWS networking.

Custom domains for applications

To use a custom hostname for a route, you must update your DNS provider by creating a canonical name (CNAME) record. Your CNAME record should map the OpenShift canonical router hostname to your custom domain. The OpenShift canonical router hostname is shown on the Route Details page after a route is created. Alternatively, a wildcard CNAME record can be created once to route all subdomains for a given hostname to the cluster’s router.

Domain validated certificates

Red Hat OpenShift Service on AWS includes TLS security certificates needed for both internal and external services on the cluster. For external routes, there are two separate TLS wildcard certificates that are provided and installed on each cluster: one is for the web console and route default hostnames, and the other is for the API endpoint. Let’s Encrypt is the certificate authority used for certificates. Routes within the cluster, such as the internal API endpoint, use TLS certificates signed by the cluster’s built-in certificate authority and require the CA bundle available in every pod for trusting the TLS certificate.

Custom certificate authorities for builds

Red Hat OpenShift Service on AWS supports the use of custom certificate authorities to be trusted by builds when pulling images from an image registry.

Load Balancers

Red Hat OpenShift Service on AWS uses up to five different load balancers:

  • An internal control plane load balancer that is internal to the cluster and used to balance traffic for internal cluster communications.

  • An external control plane load balancer that is used for accessing the OpenShift and Kubernetes APIs. This load balancer can be disabled in OCM. If this load balancer is disabled, Red Hat reconfigures the API DNS to point to the internal control plane load balancer.

  • An external control plane load balancer for Red Hat that is reserved for cluster management by Red Hat. Access is strictly controlled, and communication is only possible from whitelisted bastion hosts.

  • A default external router/ingress load balancer that is the default application load balancer, denoted by apps in the URL. The default load balancer can be configured in OCM to be either publicly accessible over the Internet or only privately accessible over a pre-existing private connection. All application routes on the cluster are exposed on this default router load balancer, including cluster services such as the logging UI, metrics API, and registry.

  • Optional: A secondary router/ingress load balancer that is a secondary application load balancer, denoted by apps2 in the URL. The secondary load balancer can be configured in OCM to be either publicly accessible over the Internet or only privately accessible over a pre-existing private connection. If a Label match is configured for this router load balancer, then only application routes matching this label are exposed on this router load balancer; otherwise, all application routes are also exposed on this router load balancer.

  • Optional: Load balancers for services. Enable non-HTTP/SNI traffic and non-standard ports for services. These load balancers can be mapped to a service running on Red Hat OpenShift Service on AWS to enable advanced ingress features, such as non-HTTP/SNI traffic or the use of non-standard ports. These can be purchased in groups of 4 for standard clusters or can be provisioned without charge in Red Hat Customer Cloud Subscription (CCS) clusters; however, each AWS account has a quota which limits the number of Classic Load Balancers that can be used within each cluster.

Cluster ingress

Project administrators can add route annotations for many different purposes, including ingress control through IP allow-listing.

Ingress policies can also be changed by using NetworkPolicy objects, which leverage the ovs-networkpolicy plug-in. This allows for full control over the ingress network policy down to the pod level, including between pods on the same cluster and even in the same namespace.

All cluster ingress traffic will go through the defined load balancers. Direct access to all nodes is blocked by cloud configuration.

Cluster egress

Pod egress traffic control through EgressNetworkPolicy objects can be used to prevent or limit outbound traffic in Red Hat OpenShift Service on AWS.

Public outbound traffic from the control plane and infrastructure nodes is required and necessary to maintain cluster image security and cluster monitoring. This requires that the 0.0.0.0/0 route belongs only to the Internet gateway; it is not possible to route this range over private connections.

OpenShift 4 clusters use NAT gateways to present a public, static IP for any public outbound traffic leaving the cluster. Each availability zone a cluster is deployed into receives a distinct NAT gateway, therefore up to 3 unique static IP addresses can exist for cluster egress traffic. Any traffic that remains inside the cluster, or that does not go out to the public Internet, will not pass through the NAT gateway and will have a source IP address belonging to the node that the traffic originated from. Node IP addresses are dynamic; therefore, a customer must not rely on whitelisting individual IP addresses when accessing private resources.

Customers can determine their public static IP addresses by running a pod on the cluster and then querying an external service. For example:

$ oc run ip-lookup --image=busybox -i -t --restart=Never --rm -- /bin/sh -c "/bin/nslookup -type=a myip.opendns.com resolver1.opendns.com | grep -E 'Address: [0-9.]+'"

Cloud network configuration

Red Hat OpenShift Service on AWS allows for the configuration of a private network connection through AWS-managed technologies:

  • VPN connections

  • VPC peering

  • Transit Gateway

  • Direct Connect

Red Hat site reliability engineers (SREs) do not monitor private network connections. Monitoring these connections is the responsibility of the customer.

DNS forwarding

For Red Hat OpenShift Service on AWS clusters that have a private cloud network configuration, a customer can specify internal DNS servers available on that private connection, that should be queried for explicitly provided domains.

Storage

This section provides information about the service definition for Red Hat OpenShift Service on AWS storage.

Encrypted-at-rest OS and node storage

Control plane nodes use encrypted-at-rest AWS Elastic Block Store (EBS) storage.

Encrypted-at-rest PV

EBS volumes that are used for PVs are encrypted-at-rest by default.

Block storage (RWO)

Persistent volumes (PVs) are backed by AWS EBS, which is Read-Write-Once.

PVs can be attached only to a single node at a time and are specific to the availability zone in which they were provisioned. However, PVs can be attached to any node in the availability zone.

Each cloud provider has its own limits for how many PVs can be attached to a single node. See AWS instance type limits for details.

Platform

This section provides information about the service definition for the Red Hat OpenShift Service on AWS platform.

Cluster backup policy

It is critical that customers have a backup plan for their applications and application data.

Application and application data backups are not a part of the Red Hat OpenShift Service on AWS service. All Kubernetes objects and persistent volumes (PVs) in each Red Hat OpenShift Service on AWS cluster are backed up to facilitate a prompt recovery in the unlikely event that a cluster becomes irreparably inoperable.

The backups are stored in a secure object storage, or multiple availability zone, bucket in the same account as the cluster. Node root volumes are not backed up, as Red Hat CoreOS is fully managed by the Red Hat OpenShift Service on AWS cluster and no stateful data should be stored on a node’s root volume.

The following table shows the frequency of backups:

Component Snapshot frequency Retention Notes

Full object store backup, all cluster PVs

Daily at 0100 UTC

7 days

This is a full backup of all Kubernetes objects, as well as all mounted PVs in the cluster.

Full object store backup, all cluster PVs

Weekly on Mondays at 0200 UTC

30 days

This is a full backup of all Kubernetes objects, as well as all mounted PVs in the cluster.

Full object store backup

Hourly at 17 minutes past the hour

24 hours

This is a full backup of all Kubernetes objects. No PVs are backed up in this backup schedule.

Autoscaling

Node autoscaling is available on Red Hat OpenShift Service on AWS.

Daemonsets

Customers can create and run daemonsets on Red Hat OpenShift Service on AWS. To restrict daemonsets to only running on worker nodes, use the following nodeSelector:

...
spec:
  nodeSelector:
    role: worker
...

Multiple availability zone

In a multiple availability zone cluster, control plane nodes are distributed across availability zones and at least one worker node is required in each availability zone.

Node labels

Custom node labels are created by Red Hat during node creation and cannot be changed on Red Hat OpenShift Service on AWS clusters at this time. However, custom labels are supported when creating new machine pools.

OpenShift version

Red Hat OpenShift Service on AWS is run as a service and is kept up to date with the latest OpenShift Container Platform version. Upgrade scheduling to the latest version is available.

Upgrades

Upgrades can be scheduled using the rosa CLI utility or through OpenShift Cluster Manager (OCM).

See the OpenShift Dedicated Life Cycle for more information on the upgrade policy and procedures.

Windows Containers

Windows Containers are not available on Red Hat OpenShift Service on AWS at this time.

Container engine

Red Hat OpenShift Service on AWS runs on OpenShift 4 and uses CRI-O as the only available container engine.

Operating system

Red Hat OpenShift Service on AWS runs on OpenShift 4 and uses Red Hat CoreOS as the operating system for all control plane and worker nodes.

Kubernetes Operator support

All Operators listed in the Operator Hub marketplace should be available for installation. These operators are considered customer workloads, and are not monitored by Red Hat SRE.

Security

This section provides information about the service definition for Red Hat OpenShift Service on AWS security.

Authentication provider

Authentication for the cluster can be configured using either the OpenShift Cluster Manager (OCM) or cluster creation process or using the rosa CLI. Red Hat OpenShift Service on AWS is not an identity provider, and all access to the cluster must be managed by the customer as part of their integrated solution. The use of multiple identity providers provisioned at the same time is supported. The following identity providers are supported:

  • GitHub or GitHub Enterprise

  • GitLab

  • Google

  • LDAP

  • OpenID Connect

Privileged containers

Privileged containers are available for users with the cluster-admin role. Usage of privileged containers as cluster-admin is subject to the responsibilities and exclusion notes in the Red Hat Enterprise Agreement Appendix 4 (Online Subscription Services).

Customer administrator user

In addition to normal users, Red Hat OpenShift Service on AWS provides access to an Red Hat OpenShift Service on AWS-specific group called dedicated-admin. Any users on the cluster that are members of the dedicated-admin group:

  • Have administrator access to all customer-created projects on the cluster.

  • Can manage resource quotas and limits on the cluster.

  • Can add and manage NetworkPolicy objects.

  • Are able to view information about specific nodes and PVs in the cluster, including scheduler information.

  • Can access the reserved dedicated-admin project on the cluster, which allows for the creation of service accounts with elevated privileges and also gives the ability to update default limits and quotas for projects on the cluster.

Cluster administration role

The administrator of Red Hat OpenShift Service on AWS has default access to the cluster-admin role for your organization’s cluster. While logged into an account with the cluster-admin role, users have increased permissions to run privileged security contexts.

Project self-service

By default, all users have the ability to create, update, and delete their projects. This can be restricted if a member of the dedicated-admin group removes the self-provisioner role from authenticated users:

$ oc adm policy remove-cluster-role-from-group self-provisioner system:authenticated:oauth

Restrictions can be reverted by applying:

$ oc adm policy add-cluster-role-to-group self-provisioner system:authenticated:oauth

Regulatory compliance

See Understanding process and security for ROSA for the latest compliance information.

Network security

With Red Hat OpenShift Service on AWS, AWS provides a standard DDoS protection on all load balancers, called AWS Shield. This provides 95% protection against most commonly used level 3 and 4 attacks on all the public facing load balancers used for Red Hat OpenShift Service on AWS. A 10-second timeout is added for HTTP requests coming to the haproxy router to receive a response or the connection is closed to provide additional protection.

Additional resources