Chapter 1. Limits and scalability
To architect a stable and fully supported OpenShift Dedicated cluster, review the information about tested cluster maximums, the test environment, and configuration. Understanding these cluster maximums and the recommended control plane and infrastructure node sizing ensures your environment is optimized for scale.
1.1. Cluster maximums
OpenShift Dedicated clusters are tested against specific object maximums. Adhering to these supported limits helps you successfully architect and deploy a scalable, reliable environment.
These guidelines are based on a cluster of 249 compute (also known as worker) nodes in a multiple availability zone configuration. For smaller clusters, the maximums are lower.
| Maximum type | 4.x tested maximum |
|---|---|
| Number of pods [1] | 25,000 |
| Number of pods per node | 250 |
| Number of pods per core | There is no default value |
| Number of namespaces [2] | 5,000 |
| Number of pods per namespace [3] | 25,000 |
| Number of services [4] | 10,000 |
| Number of services per namespace | 5,000 |
| Number of back ends per service | 5,000 |
| Number of deployments per namespace [3] | 2,000 |
- The pod count displayed here is the number of test pods. The actual number of pods depends on the memory, CPU, and storage requirements of the application.
- When there are a large number of active projects, etcd can suffer from poor performance if the keyspace grows excessively large and exceeds the space quota. Periodic maintenance of etcd, including defragmentation, is highly recommended to make etcd storage available.
- There are several control loops in the system that must iterate over all objects in a given namespace as a reaction to some changes in state. Having a large number of objects of a type, in a single namespace, can make those loops expensive and slow down processing the state changes. The limit assumes that the system has enough CPU, memory, and disk to satisfy the application requirements.
-
Each service port and each service back end has a corresponding entry in
iptables. The number of back ends of a given service impacts the size of the endpoints objects, which then impacts the size of data sent throughout the system.
1.2. OpenShift Dedicated testing environment and configuration
The following OpenShift Dedicated environment and configuration settings were tested on the AWS cloud platform. Adhering to these cluster maximums ensures your environment is fully supported and optimized for scale.
| Node | Type | vCPU | RAM(GiB) | Disk type | Disk size(GiB)/IOPS | Count | Region |
|---|---|---|---|---|---|---|---|
| Control plane/etcd [1] | m5.4xlarge | 16 | 64 | gp3 | 350 / 1,000 | 3 | us-west-2 |
| Infrastructure nodes [2] | r5.2xlarge | 8 | 64 | gp3 | 300 / 900 | 3 | us-west-2 |
| Workload [3] | m5.2xlarge | 8 | 32 | gp3 | 350 / 900 | 3 | us-west-2 |
| Compute nodes | m5.2xlarge | 8 | 32 | gp3 | 350 / 900 | 102 | us-west-2 |
- io1 disks are used for control plane/etcd nodes in all versions earlier than 4.10.
- Infrastructure nodes are used to host monitoring components because Prometheus can claim a large amount of memory, depending on usage patterns.
- Workload nodes are dedicated to run performance and scalability workload generators.
Larger cluster sizes and higher object counts might be reachable. However, the sizing of the infrastructure nodes limits the amount of memory that is available to Prometheus. When creating, modifying, or deleting objects, Prometheus stores the metrics in its memory for roughly 3 hours before persisting the metrics on disk. If the rate of creation, modification, or deletion of objects is too high, Prometheus can become overwhelmed and fail due to the lack of memory resources.
1.3. Control plane and infrastructure node sizing and scaling
When you install an OpenShift Dedicated cluster, the sizing of the control plane and infrastructure nodes is automatically determined by the compute node count.
To maintain cluster stability, the Red Hat Site Reliability Engineering (SRE) team automatically adjusts your control plane and infrastructure nodes whenever you change your compute node count.
1.3.1. Node sizing during installation
During the installation process, the sizing of the control plane and infrastructure nodes is dynamically calculated. The sizing calculation is based on the number of compute nodes in a cluster.
The following tables list the control plane and infrastructure node sizing that is applied during installation.
AWS control plane and infrastructure node size:
| Number of compute nodes | Control plane size | Infrastructure node size |
|---|---|---|
| 1 to 25 | m5.2xlarge | r5.xlarge |
| 26 to 100 | m5.4xlarge | r5.2xlarge |
| 101 to 249 | m5.8xlarge | r5.4xlarge |
Google Cloud control plane and infrastructure node size:
| Number of compute nodes | Control plane size | Infrastructure node size |
|---|---|---|
| 1 to 25 | custom-8-32768 | custom-4-32768-ext |
| 26 to 100 | custom-16-65536 | custom-8-65536-ext |
| 101 to 249 | custom-32-131072 | custom-16-131072-ext |
Google Cloud control plane and infrastructure node size for clusters created on or after 21 June 2024:
| Number of compute nodes | Control plane size | Infrastructure node size |
|---|---|---|
| 1 to 25 | n2-standard-8 | n2-highmem-4 |
| 26 to 100 | n2-standard-16 | n2-highmem-8 |
| 101 to 249 | n2-standard-32 | n2-highmem-16 |
The maximum number of compute nodes on OpenShift Dedicated clusters version 4.14.14 and later is 249. For earlier versions, the limit is 180.
1.3.2. Node scaling after installation
If you change the number of compute nodes after installation, the control plane and infrastructure nodes are scaled by the Red Hat Site Reliability Engineering (SRE) team as required. The nodes are scaled to maintain platform stability.
Postinstallation scaling requirements for control plane and infrastructure nodes are assessed on a case-by-case basis. Node resource consumption and received alerts are taken into consideration.
Rules for control plane node resizing alerts
The resizing alert is triggered for the control plane nodes in a cluster when the following occurs:
Control plane nodes sustain over 66% usage on average in a cluster.
NoteThe maximum number of compute nodes on OpenShift Dedicated is 180.
Rules for infrastructure node resizing alerts
Resizing alerts are triggered for the infrastructure nodes in a cluster when it has high-sustained CPU or memory usage. This high-sustained usage status is:
- Infrastructure nodes sustain over 50% usage on average in a cluster with a single availability zone using 2 infrastructure nodes.
Infrastructure nodes sustain over 66% usage on average in a cluster with multiple availability zones using 3 infrastructure nodes.
NoteThe maximum number of compute nodes on OpenShift Dedicated cluster versions 4.14.14 and later is 249. For earlier versions, the limit is 180.
The resizing alerts only appear after sustained periods of high usage. Short usage spikes, such as a node temporarily going down causing the other node to scale up, do not trigger these alerts.
The SRE team might scale the control plane and infrastructure nodes for additional reasons, for example to manage an increase in resource consumption on the nodes.
1.3.3. Sizing considerations for larger clusters
For larger clusters, infrastructure node sizing can become a significant impacting factor to scalability. There are many factors that influence the stated thresholds, including the etcd version or storage data format.
Exceeding these limits does not necessarily mean that the cluster will fail. In most cases, exceeding these numbers results in lower overall performance.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}