spec:
additionalKernelArgs:
- "rcupdate.rcu_normal_after_boot=0"
- "efi=runtime"
Before you can deploy virtual distributed unit (vDU) applications, you need to tune and configure the cluster host firmware and various other cluster configuration settings. Use the following information to validate the cluster configuration to support vDU workloads.
For more information about single-node OpenShift clusters tuned for vDU application deployments, see Reference configuration for deploying vDUs on single-node OpenShift.
Use the following table as the basis to configure the cluster host firmware for vDU applications running on OpenShift Container Platform 4.11.
The following table is a general recommendation for vDU cluster host firmware configuration. Exact firmware settings will depend on your requirements and specific hardware platform. Automatic setting of firmware is not handled by the zero touch provisioning pipeline. |
Firmware setting | Configuration | Description |
---|---|---|
HyperTransport (HT) |
Enabled |
HyperTransport (HT) bus is a bus technology developed by AMD. HT provides a high-speed link between the components in the host memory and other system peripherals. |
UEFI |
Enabled |
Enable booting from UEFI for the vDU host. |
CPU Power and Performance Policy |
Performance |
Set CPU Power and Performance Policy to optimize the system for performance over energy efficiency. |
Uncore Frequency Scaling |
Disabled |
Disable Uncore Frequency Scaling to prevent the voltage and frequency of non-core parts of the CPU from being set independently. |
Uncore Frequency |
Maximum |
Sets the non-core parts of the CPU such as cache and memory controller to their maximum possible frequency of operation. |
Performance P-limit |
Disabled |
Disable Performance P-limit to prevent the Uncore frequency coordination of processors. |
Enhanced Intel® SpeedStep Tech |
Enabled |
Enable Enhanced Intel SpeedStep to allow the system to dynamically adjust processor voltage and core frequency that decreases power consumption and heat production in the host. |
Intel® Turbo Boost Technology |
Enabled |
Enable Turbo Boost Technology for Intel-based CPUs to automatically allow processor cores to run faster than the rated operating frequency if they are operating below power, current, and temperature specification limits. |
Intel Configurable TDP |
Enabled |
Enables Thermal Design Power (TDP) for the CPU. |
Configurable TDP Level |
Level 2 |
TDP level sets the CPU power consumption required for a particular performance rating. TDP level 2 sets the CPU to the most stable performance level at the cost of power consumption. |
Energy Efficient Turbo |
Disabled |
Disable Energy Efficient Turbo to prevent the processor from using an energy-efficiency based policy. |
Hardware P-States |
Disabled |
Disable |
Package C-State |
C0/C1 state |
Use C0 or C1 states to set the processor to a fully active state (C0) or to stop CPU internal clocks running in software (C1). |
C1E |
Disabled |
CPU Enhanced Halt (C1E) is a power saving feature in Intel chips. Disabling C1E prevents the operating system from sending a halt command to the CPU when inactive. |
Processor C6 |
Disabled |
C6 power-saving is a CPU feature that automatically disables idle CPU cores and cache. Disabling C6 improves system performance. |
Sub-NUMA Clustering |
Disabled |
Sub-NUMA clustering divides the processor cores, cache, and memory into multiple NUMA domains. Disabling this option can increase performance for latency-sensitive workloads. |
Enable global SR-IOV and VT-d settings in the firmware for the host. These settings are relevant to bare-metal environments. |
Clusters running virtualized distributed unit (vDU) applications require a highly tuned and optimized configuration. The following information describes the various elements that you require to support vDU workloads in OpenShift Container Platform 4.11 clusters.
Check that the MachineConfig
custom resources (CRs) that you extract from the ztp-site-generate
container are applied in the cluster. The CRs can be found in the extracted out/source-crs/extra-manifest/
folder.
The following MachineConfig
CRs from the ztp-site-generate
container configure the cluster host:
CR filename | Description |
---|---|
|
Configures workload partitioning for the cluster. Apply this |
|
Loads the SCTP kernel module. These |
|
Configures the container mount namespace and Kubelet configuration. |
|
Configures accelerated startup for the cluster. |
|
Configures |
The following Operators are required for clusters running virtualized distributed unit (vDU) applications and are a part of the baseline reference configuration:
Node Tuning Operator (NTO). NTO packages functionality that was previously delivered with the Performance Addon Operator, which is now a part of NTO.
PTP Operator
SR-IOV Network Operator
Red Hat OpenShift Logging Operator
Local Storage Operator
Always use the latest supported real-time kernel version in your cluster. Ensure that you apply the following configurations in the cluster:
Ensure that the following additionalKernelArgs
are set in the cluster performance profile:
spec:
additionalKernelArgs:
- "rcupdate.rcu_normal_after_boot=0"
- "efi=runtime"
Ensure that the performance-patch
profile in the Tuned
CR configures the correct CPU isolation set that matches the isolated
CPU set in the related PerformanceProfile
CR, for example:
spec:
profile:
- name: performance-patch
# The 'include' line must match the associated PerformanceProfile name
# And the cmdline_crash CPU set must match the 'isolated' set in the associated PerformanceProfile
data: |
[main]
summary=Configuration changes profile inherited from performance created tuned
include=openshift-node-performance-openshift-node-performance-profile
[bootloader]
cmdline_crash=nohz_full=2-51,54-103 (1)
[sysctl]
kernel.timer_migration=1
[scheduler]
group.ice-ptp=0:f:10:*:ice-ptp.*
[service]
service.stalld=start,enable
service.chronyd=stop,disable
1 | Listed CPUs depend on the host hardware configuration, specifically the number of available CPUs in the system and the CPU topology. |
Always use the latest version of the realtime kernel in your OpenShift Container Platform clusters. If you are unsure about the kernel version that is in use in the cluster, you can compare the current realtime kernel version to the release version with the following procedure.
You have installed the OpenShift CLI (oc
).
You are logged in as a user with cluster-admin
privileges.
You have installed podman
.
Run the following command to get the cluster version:
$ OCP_VERSION=$(oc get clusterversion version -o jsonpath='{.status.desired.version}{"\n"}')
Get the release image SHA number:
$ DTK_IMAGE=$(oc adm release info --image-for=driver-toolkit quay.io/openshift-release-dev/ocp-release:$OCP_VERSION-x86_64)
Run the release image container and extract the kernel version that is packaged with cluster’s current release:
$ podman run --rm $DTK_IMAGE rpm -qa | grep 'kernel-rt-core-' | sed 's#kernel-rt-core-##'
4.18.0-305.49.1.rt7.121.el8_4.x86_64
This is the default realtime kernel version that ships with the release.
The realtime kernel is denoted by the string |
Check that the kernel version listed for the cluster’s current release matches actual realtime kernel that is running in the cluster. Run the following commands to check the running realtime kernel version:
Open a remote shell connection to the cluster node:
$ oc debug node/<node_name>
Check the realtime kernel version:
sh-4.4# uname -r
4.18.0-305.49.1.rt7.121.el8_4.x86_64
You can check that clusters are running the correct configuration. The following procedure describes how to check the various configurations that you require to deploy a DU application in OpenShift Container Platform 4.11 clusters.
You have deployed a cluster and tuned it for vDU workloads.
You have installed the OpenShift CLI (oc
).
You have logged in as a user with cluster-admin
privileges.
Check that the default OperatorHub sources are disabled. Run the following command:
$ oc get operatorhub cluster -o yaml
spec:
disableAllDefaultSources: true
Check that all required CatalogSource
resources are annotated for workload partitioning (PreferredDuringScheduling
) by running the following command:
$ oc get catalogsource -A -o jsonpath='{range .items[*]}{.metadata.name}{" -- "}{.metadata.annotations.target\.workload\.openshift\.io/management}{"\n"}{end}'
certified-operators -- {"effect": "PreferredDuringScheduling"}
community-operators -- {"effect": "PreferredDuringScheduling"}
ran-operators (1)
redhat-marketplace -- {"effect": "PreferredDuringScheduling"}
redhat-operators -- {"effect": "PreferredDuringScheduling"}
1 | CatalogSource resources that are not annotated are also returned. In this example, the ran-operators CatalogSource resource is not annotated and does not have the PreferredDuringScheduling annotation. |
In a properly configured vDU cluster, only a single annotated catalog source is listed. |
Check that all applicable OpenShift Container Platform Operator namespaces are annotated for workload partitioning. This includes all Operators installed with core OpenShift Container Platform and the set of additional Operators included in the reference DU tuning configuration. Run the following command: