This document details the Red Hat responsibilities for the managed Red Hat OpenShift Service on AWS (ROSA).

Acronyms and terms
  • AWS - Amazon Web Services

  • CEE - Customer Experience and Engagement (Red Hat Support)

  • CI/CD - Continuous Integration / Continuous Delivery

  • CVE - Common Vulnerabilities and Exposures

  • OCM - OpenShift Cluster Manager

  • PVs - Persistent Volumes

  • ROSA - Red Hat OpenShift Service on AWS

  • SRE - Red Hat Site Reliability Engineering

  • VPC - Virtual Private Cloud

Incident and operations management

This documentation details the Red Hat responsibilities for the Red Hat OpenShift Service on AWS (ROSA) managed service.

Platform monitoring

Red Hat site reliability engineers (SREs) maintain a centralized monitoring and alerting system for all ROSA cluster components, the SRE services, and underlying AWS accounts. Platform audit logs are securely forwarded to a centralized security information and event monitoring (SIEM) system, where they may trigger configured alerts to the SRE team and are also subject to manual review. Audit logs are retained in the SIEM system for one year. Audit logs for a given cluster are not deleted at the time the cluster is deleted.

Incident management

An incident is an event that results in a degradation or outage of one or more Red Hat services. An incident can be raised by a customer or a Customer Experience and Engagement (CEE) member through a support case, directly by the centralized monitoring and alerting system, or directly by a member of the SRE team.

Depending on the impact on the service and customer, the incident is categorized in terms of severity.

When managing a new incident, Red Hat uses the following general workflow:

  1. An SRE first responder is alerted to a new incident and begins an initial investigation.

  2. After the initial investigation, the incident is assigned an incident lead, who coordinates the recovery efforts.

  3. An incident lead manages all communication and coordination around recovery, including any relevant notifications and support case updates.

  4. The incident is recovered.

  5. The incident is documented and a root cause analysis (RCA) is performed within 3 business days of the incident.

  6. An RCA draft document will be shared with the customer within 7 business days of the incident.

Notifications

Platform notifications are configured using email. Some customer notifications are also sent to an account’s corresponding Red Hat account team, including a Technical Account Manager, if applicable.

The following activities can trigger notifications:

  • Platform incident

  • Performance degradation

  • Cluster capacity warnings

  • Critical vulnerabilities and resolution

  • Upgrade scheduling

Backup and recovery

All Red Hat OpenShift Service on AWS clusters are backed up using AWS snapshots. Notably, this does not include customer data stored on persistent volumes (PVs). All snapshots are taken using the appropriate AWS snapshot APIs and are uploaded to a secure AWS S3 object storage bucket in the same account as the cluster.

Component Snapshot frequency Retention Notes

Full object store backup, all SRE-managed cluster PVs

Daily

7 days

This is a full backup of all Kubernetes objects, such as etcd, and all SRE-managed PVs in the cluster.

Weekly

30 days

Full object store backup

Hourly

24-hour

This is a full backup of all Kubernetes objects, such as etcd. No PVs are backed up in this backup schedule.

Node root volume

Never

N/A

Nodes are considered to be short-term. Do not store anything critical on a node’s root volume.

  • The SRE rehearses recovery processes quarterly.

  • Red Hat does not commit to any Recovery Point Objective (RPO) or Recovery Time Objective (RTO).

  • Customers are responsible for taking regular backups of their data.

  • Backups performed by the SRE are taken as a precautionary measure only. They are stored in the same region as the cluster.

  • Customers can access the SRE backup data on request through a support case.

  • Red Hat encourages customers to deploy multiple availability zone (multi-AZ) clusters with workloads that follow Kubernetes best practices to ensure high availability within a region.

  • In the event an entire AWS region is unavailable, customers must install a new cluster in a different region and restore their apps using their backup data.

Cluster capacity

Evaluating and managing cluster capacity is a responsibility that is shared between Red Hat and the customer. Red Hat SRE is responsible for the capacity of all control plane and infrastructure nodes on the cluster.

Red Hat SRE also evaluates cluster capacity during upgrades and in response to cluster alerts. The impact of a cluster upgrade on capacity is evaluated as part of the upgrade testing process to ensure that capacity is not negatively impacted by new additions to the cluster. During a cluster upgrade, additional worker nodes are added to make sure that total cluster capacity is maintained during the upgrade process.

Capacity evaluations by the Red Hat SRE staff also happen in response to alerts from the cluster, after usage thresholds are exceeded for a certain period of time. Such alerts can also result in a notification to the customer.

Change management

This section describes the policies about how cluster changes, configuration changes, patches, and releases are managed.

Cluster changes are initiated in one of two ways:

  1. A customer initiates changes through self-service capabilities such as cluster deployment, worker node scaling, or cluster deletion.

  2. Red Hat site reliability engineering (SRE) initiates a change through Operator-driven capabilities such as configuration, upgrade, patching, or configuration changes.

Change history is captured in the Cluster History section in the OpenShift Cluster Manager (OCM) Overview tab and is available to customers. The change history includes, but is not limited to, logs from the following changes:

  • Adding or removing identity providers

  • Adding or removing users to or from the dedicated-admins group

  • Scaling the cluster compute nodes

  • Scaling the cluster load balancer

  • Scaling the cluster persistent storage

  • Upgrading the cluster

The SRE-initiated changes that require manual intervention by SRE generally follow this process:

  • Preparing for change

    • Change characteristics are identified and a gap analysis is performed against current state.

    • Change steps are documented and validated.

    • A communication plan and schedule are shared with all stakeholders.

    • CI/CD and end-to-end tests are updated to automate change validation.

    • A change request that captures change details is submitted for management approval.

  • Managing change

    • Automated nightly CI/CD jobs pick up the change and run tests.

    • The change is made to integration and stage environments, and manually validated before updating the customer cluster.

    • Major change notifications are sent before and after the event.

  • Reinforcing the change

    • Feedback on the change is collected and analyzed.

    • Potential gaps are diagnosed to understand resistance and automate similar change requests.

    • Corrective actions are implemented.

SRE only uses manual changes as a fallback process because manual intervention is considered to be a failure of change management.

Configuration management

The infrastructure and configuration of the Red Hat OpenShift Service on AWS environment is managed as code. SRE manages changes to the Red Hat OpenShift Service on AWS environment using a GitOps workflow and automated CI/CD pipeline.

Each proposed change undergoes a series of automated verifications immediately upon check-in. Changes are then deployed to a staging environment where they undergo automated integration testing. Finally, changes are deployed to the production environment. Each step is fully automated.

An authorized SRE reviewer must approve advancement to each step. The reviewer cannot be the same individual who proposed the change. All changes and approvals are fully auditable as part of the GitOps workflow.

Patch management

OpenShift Container Platform software and the underlying immutable Red Hat CoreOS (RHCOS) operating system image are patched for bugs and vulnerabilities in regular z-stream upgrades. Read more about RHCOS architecture in the OpenShift Container Platform documentation.

Release management

ROSA clusters can be configured for automatic upgrades on a schedule. Alternatively, you can perform manual upgrades using the rosa CLI. For more details, see the Life Cycle policy.

Customers can review the history of all cluster upgrade events in their OCM web console on the Events tab.

Identity and access management

Most access by Red Hat site reliability engineering (SRE) teams is done using cluster Operators through automated configuration management.

SRE access to all Red Hat OpenShift Service on AWS clusters

SREs access Red Hat OpenShift Service on AWS clusters through the web console or command-line tools. Authentication requires multi-factor authentication (MFA) with industry-standard requirements for password complexity and account lockouts. SREs must authenticate as individuals to ensure auditability. All authentication attempts are logged to a Security Information and Event Management (SIEM) system.

SREs access private clusters using an encrypted tunnel through a hardened SRE support pod running in the cluster. Connections to the SRE support pod are permitted only from a secured Red Hat network using an IP allow-list. In addition to the cluster authentication controls described above, authentication to the SRE support pod is controlled by using SSH keys. SSH key authorization is limited to SRE staff and automatically synchronized with Red Hat corporate directory data. Corporate directory data is secured and controlled by HR systems, including management review, approval, and audits.

Privileged access controls in Red Hat OpenShift Service on AWS

SRE adheres to the principle of least privilege when accessing Red Hat OpenShift Service on AWS and AWS components. There are four basic categories of manual SRE access:

  • SRE admin access through the Red Hat Portal with normal two-factor authentication and no privileged elevation.

  • SRE admin access through the Red Hat corporate SSO with normal two-factor authentication and no privileged elevation.

  • OpenShift elevation, which is a manual elevation using Red Hat SSO. Access is limited to 2 hours, is fully audited, and requires management approval.

  • AWS access or elevation, which is a manual elevation for AWS console access. Access is limited to 60 minutes, is fully audited, and requires management approval.

Each of these access types have different levels of access to components:

Component Typical SRE admin access (Red Hat Portal) Typical SRE admin access (Red Hat SSO) Openshift elevation Cloud provider access or elevation

OpenShift Cluster Manager (OCM)

R/W

No access

No access

No access

OpenShift console

No access

R/W

R/W

No access

Node Operatiing system

No access

A specific list of elevated OS and network permissions.

A specific list of elevated OS and network permissions.

No access

AWS Console

No access

No access, but this is the account used to request cloud provider access.

No access

All cloud provider permissions using the SRE identity.

SRE access to AWS accounts

Red Hat personnel do not access AWS accounts in the course of routine Red Hat OpenShift Service on AWS operations. For emergency troubleshooting purposes, the SREs have well-defined and auditable procedures to access cloud infrastructure accounts.

SREs generate a short-lived AWS access token for the osdManagedAdminSRE user using the AWS Security Token Service (STS). Access to the STS token is audit-logged and traceable back to individual users. The osdManagedAdminSRE user has the AdministratorAccess IAM policy attached.

Red Hat support access

Members of the Red Hat Customer Experience and Engagement (CEE) team typically have read-only access to parts of the cluster. Specifically, CEE has limited access to the core and product namespaces and does not have access to the customer namespaces.

Role Core namespace Layered product namespace Customer namespace AWS account*

OpenShift SRE

Read: All

Write: Very

limited [1]

Read: All

Write: None

Read: None[2]

Write: None

Read: All [3]

Write: All [3]

CEE

Read: All

Write: None

Read: All

Write: None

Read: None[2]

Write: None

Read: None

Write: None

Customer administrator

Read: None

Write: None

Read: None

Write: None

Read: All

Write: All

Read: All

Write: All

Customer user

Read: None

Write: None

Read: None

Write: None

Read: Limited[4]

Write: Limited[4]

Read: None

Write: None

Everybody else

Read: None

Write: None

Read: None

Write: None

Read: None

Write: None

Read: None

Write: None

  1. Limited to addressing common use cases such as failing deployments, upgrading a cluster, and replacing bad worker nodes.

  2. Red Hat associates have no access to customer data by default.

  3. SRE access to the AWS account is an emergency procedure for exceptional troubleshooting during a documented incident.

  4. Limited to what is granted through RBAC by the Customer Administrator, as well as namespaces created by the user.

Customer access

Customer access is limited to namespaces created by the customer and permissions that are granted using RBAC by the Customer Administrator role. Access to the underlying infrastructure or product namespaces is generally not permitted without cluster-admin access. More information on customer access and authentication can be found in the "Understanding Authentication" section of the documentation.

Access approval and review

New SRE user access requires management approval. Separated or transferred SRE accounts are removed as authorized users through an automated process. Additionally, the SRE performs periodic access review, including management sign-off of authorized user lists.

Security and regulation compliance

Security and regulation compliance includes tasks such as the implementation of security controls and compliance certification.

Data classification

Red Hat defines and follows a data classification standard to determine the sensitivity of data and highlight inherent risk to the confidentiality and integrity of that data while it is collected, used, transmitted, stored, and processed. Customer-owned data is classified at the highest level of sensitivity and handling requirements.

Data management

Red Hat OpenShift Service on AWS (ROSA) uses AWS KMS to help securely manage keys for encrypted data. These keys are used for control plane data volumes that are encrypted by default. Persistent volumes (PVs) for customer applications also use AWS KMS for key management.

When a customer deletes their ROSA cluster, all cluster data is permanently deleted, including control plane data volumes, customer application data volumes, such as PVs, and backup data.

Vulnerability management

Red Hat performs periodic vulnerability scanning of ROSA using industry standard tools. Identified vulnerabilities are tracked to their remediation according to timelines based on severity. Vulnerability scanning and remediation activities are documented for verification by third-party assessors in the course of compliance certification audits.

Network security

Firewall and DDoS protection

Each ROSA cluster is protected by a secure network configuration using firewall rules for AWS Security Groups. ROSA customers are also protected against DDoS attacks with AWS Shield Standard.

Private clusters and network connectivity

Customers can optionally configure their ROSA cluster endpoints, such as web console, API, and application router, to be made private so that the cluster control plane and applications are not accessible from the Internet. Red Hat SRE still requires Internet-accessible endpoints that are protected with IP allow-lists.

AWS customers can configure a private network connection to their ROSA cluster through technologies such as AWS VPC peering, AWS VPN, or AWS Direct Connect.

Cluster network access controls

Fine-grained network access control rules can be configured by customers, on a per-project basis, using NetworkPolicy objects and the OpenShift SDN.

Penetration testing

Red Hat performs periodic penetration tests against ROSA. Tests are performed by an independent internal team by using industry standard tools and best practices.

Any issues that may be discovered are prioritized based on severity. Any issues found belonging to open source projects are shared with the community for resolution.

Compliance

ROSA follows common industry best practices for security and controls.

ROSA is certified for SOC 2 Type I and ISO 27001.

Disaster recovery

Red Hat OpenShift Service on AWS (ROSA) provides disaster recovery for failures that occur at the pod, worker node, infrastructure node, master node, and availability zone levels.

All disaster recovery requires that the customer use best practices for deploying highly available applications, storage, and cluster architecture, such as single-zone deployment or multi-zone deployment, to account for the level of desired availability.

One single-zone cluster will not provide disaster avoidance or recovery in the event of an availability zone or region outage. Multiple single-zone clusters with customer-maintained failover can account for outages at the zone or at the regional level.

One multi-zone cluster will not provide disaster avoidance or recovery in the event of a full region outage. Multiple multi-zone clusters with customer-maintained failover can account for outages at the regional level.

Additional resources