Telco core reference design components

The operating system needs a certain amount of CPU to perform all the support tasks including kernel networking.

A system with Hyper-Threading enabled must always put all core sibling threads to the same pool of CPUs.

The set of reserved and isolated cores must include all CPU cores.

Core 0 of each NUMA node must be included in the reserved CPU set.

Isolated cores might be impacted by interrupts. The following annotations must be attached to the pod if guaranteed QoS pods require full use of the CPU:

cpu-load-balancing.crio.io: "disable"
cpu-quota.crio.io: "disable"
irq-load-balancing.crio.io: "disable"

When per-pod power management is enabled with PerformanceProfile.workloadHints.perPodPowerManagement the following annotations must also be attached to the pod if guaranteed QoS pods require full use of the CPU:

cpu-c-states.crio.io: "disable"
cpu-freq-governor.crio.io: "performance"

The minimum reserved capacity (systemReserved) required can be found by following the guidance in "Which amount of CPU and memory are recommended to reserve for the system in OCP 4 nodes?"

The actual required reserved CPU capacity depends on the cluster configuration and workload attributes.

This reserved CPU value must be rounded up to a full core (2 hyper-thread) alignment.

Changes to the CPU partitioning will drain and reboot the nodes in the MCP.

The reserved CPUs reduce the pod density, as the reserved CPUs are removed from the allocatable capacity of the OpenShift node.

The real-time workload hint should be enabled if the workload is real-time capable.

Hardware without Interrupt Request (IRQ) affinity support will impact isolated CPUs. To ensure that pods with guaranteed CPU QoS have full use of allocated CPU, all hardware in the server must support IRQ affinity.

The impact of cluster CPU use is based on the number or size of logs generated and the amount of log filtering configured.

The reference configuration does not include shipping of application logs. Inclusion of application logs in the configuration requires evaluation of the application logging rate and sufficient additional CPU resources allocated to the reserved set.

You can specify a maximum latency that is C-state for a low latency pod when using per-pod power management. Previously, C-states could only be disabled completely on a per pod basis.

Power configuration relies on appropriate BIOS configuration, for example, enabling C-states and P-states. Configuration varies between hardware vendors.

Latency: To ensure that latency-sensitive workloads meet their requirements, you will need either a high-power configuration or a per-pod power management configuration. Per-pod power management is only available for Guaranteed QoS Pods with dedicated pinned CPUs.

Pod CPU and memory metrics and alerts for user workloads

Monitoring configuration must enable the dedicated service monitor feature for accurate representation of pod metrics

The Prometheus retention period is specified by the user. The value used is a tradeoff between operational requirements for maintaining historical data on the cluster against CPU and storage resources. Longer retention periods increase the need for storage and require additional CPU to manage the indexing of data.

NUMA-aware scheduling with the NUMA Resources Operator is now generally available in OpenShift Container Platform 4.14.

With this release, you can exclude advertising the Non-Uniform Memory Access (NUMA) node for the SR-IOV network to the Topology Manager. By not advertising the NUMA node for the SR-IOV network, you can permit more flexible SR-IOV network deployments during NUMA-aware pod scheduling. To exclude advertising the NUMA node for the SR-IOV network resource to the Topology Manager, set the value excludeTopology to true in the SriovNetworkNodePolicy CR. For more information, see Exclude the SR-IOV network topology for NUMA-aware scheduling.

The scheduler is a cluster-wide component responsible for selecting the right node for a given workload. It is a core part of the platform and does not require any specific configuration in the common deployment scenarios. However, there are few specific use cases described in the following section.

The default scheduler does not understand the NUMA locality of workloads. It only knows about the sum of all free resources on a worker node. This might cause workloads to be rejected when scheduled to a node with Topology manager policy set to single-numa-node or restricted.

All machine config pools must have consistent hardware configuration for example all nodes are expected to have the same NUMA zone count.

Pods might require annotations for correct scheduling and isolation. For more information on annotations, see the "CPU Partitioning and performance tuning" section.

Installation from declarative CRs

Does not require additional servers to support installation

Supports install in disconnected environment

Disconnected installation requires a reachable registry with all required content mirrored.

Networking configuration should be applied as NMState configuration during installation in preference to day-2 configuration by using the NMState Operator.

DPDK applications that need to inject traffic to the kernel can run in non-privileged pods with the help of the TAP CNI plugin. Furthermore, in this 4.14 release that ability to create a MAC-VLAN, IP-VLAN, and VLAN subinterface based on a master interface in a container namespace is generally available.

SecurityContextConstraints: All workload pods should be run with restricted-v2 or restricted SCC.

Seccomp: All pods should be run with the RuntimeDefault (or stronger) seccomp profile.

Rootless DPDK pods: Many user-plane networking (DPDK) CNFs require pods to run with root privileges. With this feature, a conformant DPDK pod can be run without requiring root privileges.

Storage: The storage network should be isolated and non-routable to other cluster networks. See the "Storage" section for additional details.

Rootless DPDK pods requires the following additional configuration steps:

For rootless DPDK pod support, the SELinux boolean container_use_devices must be enabled on the host for the TAP device to be created. This introduces a security risk that is acceptable for short to mid-term use. Other solutions will be explored.

Cluster scales to at least 120 nodes