Starting on March 12, 2025, OpenShift docs will only be available at docs.redhat.com. From that time on, docs.openshift.com links will automatically redirect to their locations on docs.redhat.com.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Telco RAN DU reference design specifications

Reference design specifications for telco RAN DU 5G deployments

Red Hat and certified partners offer deep technical expertise and support for networking and operational capabilities required to run telco applications on OpenShift Container Platform 4.18 clusters.

Red Hat’s telco partners require a well-integrated, well-tested, and stable environment that can be replicated at scale for enterprise 5G solutions. The telco core and RAN DU reference design specifications (RDS) outline the recommended solution architecture based on a specific version of OpenShift Container Platform. Each RDS describes a tested and validated platform configuration for telco core and RAN DU use models. The RDS ensures an optimal experience when running your applications by defining the set of critical KPIs for telco 5G core and RAN DU. Following the RDS minimizes high severity escalations and improves application stability.

5G use cases are evolving and your workloads are continually changing. Red Hat is committed to iterating over the telco core and RAN DU RDS to support evolving requirements based on customer and partner feedback.

The reference configuration includes the configuration of the far edge clusters and hub cluster components.

The reference configurations in this document are deployed using a centrally managed hub cluster infrastructure as shown in the following image.

A diagram showing two distinctive network far edge deployment processes
Figure 1. Telco RAN DU deployment architecture

Reference design scope

The telco core and telco RAN reference design specifications (RDS) capture the recommended, tested, and supported configurations to get reliable and repeatable performance for clusters running the telco core and telco RAN profiles.

Each RDS includes the released features and supported configurations that are engineered and validated for clusters to run the individual profiles. The configurations provide a baseline OpenShift Container Platform installation that meets feature and KPI targets. Each RDS also describes expected variations for each individual configuration. Validation of each RDS includes many long duration and at-scale tests.

The validated reference configurations are updated for each major Y-stream release of OpenShift Container Platform. Z-stream patch releases are periodically re-tested against the reference configurations.

Deviations from the reference design

Deviating from the validated telco core and telco RAN DU reference design specifications (RDS) can have significant impact beyond the specific component or feature that you change. Deviations require analysis and engineering in the context of the complete solution.

All deviations from the RDS should be analyzed and documented with clear action tracking information. Due diligence is expected from partners to understand how to bring deviations into line with the reference design. This might require partners to provide additional resources to engage with Red Hat to work towards enabling their use case to achieve a best in class outcome with the platform. This is critical for the supportability of the solution and ensuring alignment across Red Hat and with partners.

Deviation from the RDS can have some or all of the following consequences:

  • It can take longer to resolve issues.

  • There is a risk of missing project service-level agreements (SLAs), project deadlines, end provider performance requirements, and so on.

  • Unapproved deviations may require escalation at executive levels.

    Red Hat prioritizes the servicing of requests for deviations based on partner engagement priorities.

Engineering considerations for the RAN DU use model

The RAN DU use model configures an OpenShift Container Platform cluster running on commodity hardware for hosting RAN distributed unit (DU) workloads. Model and system level considerations are described below. Specific limits, requirements and engineering considerations for individual components are detailed in later sections.

For details of the RAN DU KPI test results, see the Telco RAN DU reference design specification KPI test results for OpenShift 4.18. This information is only available to customers and partners.
Workloads

  • DU workloads are described in "Telco RAN DU application workloads".

  • DU worker nodes are Intel 3rd Generation Xeon (IceLake) 2.20 GHz or better with host firmware tuned for maximum performance.

Resources

The maximum number of running pods in the system, inclusive of application workload and OpenShift Container Platform pods, is 120.

Resource utilization

OpenShift Container Platform resource utilization varies depending on many factors such as the following application workload characteristics:

  • Pod count

  • Type and frequency of probes

  • Messaging rates on the primary or secondary CNI with kernel networking

  • API access rate

  • Logging rates

  • Storage IOPS

Resource utilization is measured for clusters configured as follows:

  1. The cluster is a single host with single-node OpenShift installed.

  2. The cluster runs the representative application workload described in "Reference application workload characteristics".

  3. The cluster is managed under the constraints detailed in "Hub cluster management characteristics".

  4. Components noted as "optional" in the use model configuration are not included.

Configuration outside the scope of the RAN DU RDS that do not meet these criteria requires additional analysis to determine the impact on resource utilization and ability to meet KPI targets. You might need to allocate additional cluster resources to meet these requirements.
Reference application workload characteristics

  1. Uses 15 pods and 30 containers for the vRAN application including its management and control functions

  2. Uses an average of 2 ConfigMap and 4 Secret CRs per pod

  3. Uses a maximum of 10 exec probes with a frequency of not less than 10 seconds

  4. Incremental application load on the kube-apiserver is less than or equal to 10% of the cluster platform usage

    You can extract CPU load can from the platform metrics. For example:

    $ query=avg_over_time(pod:container_cpu_usage:sum{namespace="openshift-kube-apiserver"}[30m])

  5. Application logs are not collected by the platform log collector

  6. Aggregate traffic on the primary CNI is less than 8 MBps

Hub cluster management characteristics

RHACM is the recommended cluster management solution and is configured to these limits:

  1. Use a maximum of 5 RHACM configuration policies with a compliant evaluation interval of not less than 10 minutes.

  2. Use a minimal number (up to 10) of managed cluster templates in cluster policies. Use hub-side templating.

  3. Disable RHACM addons with the exception of the policyController and configure observability with the default configuration.

The following table describes resource utilization under reference application load.

Table 1. Resource utilization under reference application load
Metric Limits Notes

OpenShift platform CPU usage

Less than 4000mc – 2 cores (4HT)

Platform CPU is pinned to reserved cores, including both hyper-threads of each reserved core. The system is engineered to 3 CPUs (3000mc) at steady-state to allow for periodic system tasks and spikes.

OpenShift Platform memory

Less than 16G

Telco RAN DU application workloads

Develop RAN DU applications that are subject to the following requirements and limitations.

Description and limits

  • Develop cloud-native network functions (CNFs) that conform to the latest version of Red Hat best practices for Kubernetes.

  • Use SR-IOV for high performance networking.

  • Use exec probes sparingly and only when no other suitable options are available.

    • Do not use exec probes if a CNF uses CPU pinning. Use other probe implementations, for example, httpGet or tcpSocket.

    • When you need to use exec probes, limit the exec probe frequency and quantity. The maximum number of exec probes must be kept below 10, and frequency must not be set to less than 10 seconds. Exec probes cause much higher CPU usage on management cores compared to other probe types because they require process forking.

      Startup probes require minimal resources during steady-state operation. The limitation on exec probes applies primarily to liveness and readiness probes.

A test workload that conforms to the dimensions of the reference DU application workload described in this specification can be found at openshift-kni/du-test-workloads.

Telco RAN DU reference design components

The following sections describe the various OpenShift Container Platform components and configurations that you use to configure and deploy clusters to run RAN DU workloads.

Diagram showing telco RAN DU RDS components
Figure 2. Telco RAN DU reference design components

Ensure that additional components you include that are not specified in the telco RAN DU profile do not affect the CPU resources allocated to workload applications.

Out of tree drivers are not supported. 5G RAN application components are not included in the RAN DU profile and must be engineered against resources (CPU) allocated to applications.

Host firmware tuning

New in this release

  • No reference design updates in this release

Description

Tune host firmware settings for optimal performance during initial cluster deployment. For more information, see "Recommended single-node OpenShift cluster configuration for vDU application workloads". Apply tuning settings in the host firmware during initial deployment. See "Managing host firmware settings with GitOps ZTP" for more information. The managed cluster host firmware settings are available on the hub cluster as individual BareMetalHost custom resources (CRs) that are created when you deploy the managed cluster with the ClusterInstance CR and GitOps ZTP.

Create the ClusterInstance CR based on the provided reference example-sno.yaml CR.
Limits and requirements

  • You must enable Hyper-Threading in the host firmware settings

Engineering considerations

  • Tune all firmware settings for maximum performance.

  • All settings are expected to be for maximum performance unless tuned for power savings.

  • You can tune host firmware for power savings at the expense of performance as required.

  • Enable secure boot. When secure boot is enabled, only signed kernel modules are loaded by the kernel. Out-of-tree drivers are not supported.

Additional resources

CPU partitioning and performance tuning

New in this release

  • No reference design updates in this release

Description

The RAN DU use model includes cluster performance tuning via PerformanceProfile CRs for low-latency performance. The PerformanceProfile CRs are reconciled by the Node Tuning Operator. The RAN DU use case requires the cluster to be tuned for low-latency performance. For more details about node tuning with the PerformanceProfile CR, see "Tuning nodes for low latency with the performance profile".

Limits and requirements

The Node Tuning Operator uses the PerformanceProfile CR to configure the cluster. You need to configure the following settings in the telco RAN DU profile PerformanceProfile CR:

  • Set a reserved cpuset of 4 or more, equating to 4 hyper-threads (2 cores) for either of the following CPUs:

    • Intel 3rd Generation Xeon (IceLake) 2.20 GHz or better CPUs with host firmware tuned for maximum performance

    • AMD EPYC Zen 4 CPUs (Genoa, Bergamo, or newer)

      AMD EPYC Zen 4 CPUs (Genoa, Bergamo, or newer) are fully supported. Power consumption evaluations are ongoing. It is recommended to evaluate features, such as per-pod power management, to determine any potential impact on performance.

  • Set the reserved cpuset to include both hyper-thread siblings for each included core. Unreserved cores are available as allocatable CPU for scheduling workloads.

  • Ensure that hyper-thread siblings are not split across reserved and isolated cores.

  • Ensure that reserved and isolated CPUs include all the threads for all cores in the CPU.

  • Include Core 0 for each NUMA node in the reserved CPU set.

  • Set the huge page size to 1G.

  • Only pin OpenShift Container Platform pods which are by default configured as part of the management workload partition to reserved cores.

Engineering considerations

  • Meeting the full performance metrics requires use of the RT kernel. If required, you can use the non-RT kernel with corresponding impact to performance.

  • The number of hugepages you configure depends on application workload requirements. Variation in this parameter is expected and allowed.

  • Variation is expected in the configuration of reserved and isolated CPU sets based on selected hardware and additional components in use on the system. The variation must still meet the specified limits.

  • Hardware without IRQ affinity support affects isolated CPUs. To ensure that pods with guaranteed whole CPU QoS have full use of allocated CPUs, all hardware in the server must support IRQ affinity.

  • When workload partitioning is enabled by setting cpuPartitioningMode to AllNodes during deployment, you must allocate enough CPUs to support the operating system, interrupts, and OpenShift Container Platform pods in the PerformanceProfile CR.

Additional resources

PTP Operator

New in this release

  • No reference design updates in this release

Description

Configure PTP in cluster nodes with PTPConfig CRs for the RAN DU use case with features like Grandmaster clock (T-GM) support via GPS, ordinary clock (OC), boundary clocks (T-BC), dual boundary clocks, high availability (HA), and optional fast event notification over HTTP. PTP ensures precise timing and reliability in the RAN environment.

Limits and requirements

  • Limited to two boundary clocks for nodes with dual NICs and HA

  • Limited to two Westport channel NIC configurations for T-GM

Engineering considerations

  • RAN DU RDS configurations are provided for ordinary clocks, boundary clocks, grandmaster clocks, and highly available dual NIC boundary clocks.

  • PTP fast event notifications use ConfigMap CRs to persist subscriber details.

  • Hierarchical event subscription as described in the O-RAN specification is not supported for PTP events.

  • Use the PTP fast events REST API v2. The PTP fast events REST API v1 is deprecated. The REST API v2 is O-RAN Release 3 compliant.

SR-IOV Operator

New in this release

  • No reference design updates in this release

Description

The SR-IOV Operator provisions and configures the SR-IOV CNI and device plugins. Both netdevice (kernel VFs) and vfio (DPDK) devices are supported and applicable to the RAN DU use models.

Limits and requirements

  • Use devices that are supported for OpenShift Container Platform. See "Supported devices".

  • SR-IOV and IOMMU enablement in host firmware settings: The SR-IOV Network Operator automatically enables IOMMU on the kernel command line.

  • SR-IOV VFs do not receive link state updates from the PF. If link down detection is required you must configure this at the protocol level.

Engineering considerations

  • SR-IOV interfaces with the vfio driver type are typically used to enable additional secondary networks for applications that require high throughput or low latency.

  • Customer variation on the configuration and number of SriovNetwork and SriovNetworkNodePolicy custom resources (CRs) is expected.

  • IOMMU kernel command line settings are applied with a MachineConfig CR at install time. This ensures that the SriovOperator CR does not cause a reboot of the node when adding them.

  • SR-IOV support for draining nodes in parallel is not applicable in a single-node OpenShift cluster.

  • You must include the SriovOperatorConfig CR in your deployment; the CR is not created automatically. This CR is included in the reference configuration policies which are applied during initial deployment.

  • In scenarios where you pin or restrict workloads to specific nodes, the SR-IOV parallel node drain feature will not result in the rescheduling of pods. In these scenarios, the SR-IOV Operator disables the parallel node drain functionality.

  • NICs which do not support firmware updates under secure boot or kernel lockdown must be pre-configured with sufficient virtual functions (VFs) to support the number of VFs needed by the application workload. For Mellanox NICs, the Mellanox vendor plugin must be disabled in the SR-IOV Network Operator. For more information, see "Configuring an SR-IOV network device".

  • To change the MTU value of a virtual function after the pod has started, do not configure the MTU field in the SriovNetworkNodePolicy CR. Instead, configure the Network Manager or use a custom systemd script to set the MTU of the physical function to an appropriate value. For example:

    # ip link set dev <physical_function> mtu 9000
Additional resources

Logging

New in this release

  • No reference design updates in this release

Description

Use logging to collect logs from the far edge node for remote analysis. The recommended log collector is Vector.

Engineering considerations

  • Handling logs beyond the infrastructure and audit logs, for example, from the application workload requires additional CPU and network bandwidth based on additional logging rate.

  • As of OpenShift Container Platform 4.14, Vector is the reference log collector. Use of fluentd in the RAN use models is deprecated.

Additional resources

SRIOV-FEC Operator

New in this release

  • No reference design updates in this release

Description

SRIOV-FEC Operator is an optional 3rd party Certified Operator supporting FEC accelerator hardware.

Limits and requirements

  • Starting with FEC Operator v2.7.0:

    • Secure boot is supported

    • vfio drivers for PFs require the usage of a vfio-token that is injected into the pods. Applications in the pod can pass the VF token to DPDK by using EAL parameter --vfio-vf-token.

Engineering considerations

  • The SRIOV-FEC Operator uses CPU cores from the isolated CPU set.

  • You can validate FEC readiness as part of the pre-checks for application deployment, for example, by extending the validation policy.

Additional resources

Lifecycle Agent

New in this release

  • No reference design updates in this release

Description

The Lifecycle Agent provides local lifecycle management services for single-node OpenShift clusters.

Limits and requirements

  • The Lifecycle Agent is not applicable in multi-node clusters or single-node OpenShift clusters with an additional worker.

  • The Lifecycle Agent requires a persistent volume that you create when installing the cluster. For descriptions of partition requirements, see "Configuring a shared container directory between ostree stateroots when using GitOps ZTP".

Additional resources

Local Storage Operator

New in this release

  • No reference design updates in this release

Description

You can create persistent volumes that can be used as PVC resources by applications with the Local Storage Operator. The number and type of PV resources that you create depends on your requirements.

Engineering considerations

  • Create backing storage for PV CRs before creating the PV. This can be a partition, a local volume, LVM volume, or full disk.

  • Refer to the device listing in LocalVolume CRs by the hardware path used to access each device to ensure correct allocation of disks and partitions, for example, /dev/disk/by-path/<id>. Logical names (for example, /dev/sda) are not guaranteed to be consistent across node reboots.

Logical Volume Manager Storage

New in this release

  • No reference design updates in this release

Description

Logical Volume Manager (LVM) Storage is an optional component. It provides dynamic provisioning of both block and file storage by creating logical volumes from local devices that can be consumed as persistent volume claim (PVC) resources by applications. Volume expansion and snapshots are also possible. An example configuration is provided in the RDS with the StorageLVMCluster.yaml file.

Limits and requirements

  • In single-node OpenShift clusters, persistent storage must be provided by either LVM Storage or local storage, not both.

  • Volume snapshots are excluded from the reference configuration.

Engineering considerations

  • LVM Storage can be used as the local storage implementation for the RAN DU use case. When LVM Storage is used as the storage solution, it replaces the Local Storage Operator, and the CPU required is assigned to the management partition as platform overhead. The reference configuration must include one of these storage solutions but not both.

  • Ensure that sufficient disks or partitions are available for storage requirements.

Workload partitioning

New in this release

  • No reference design updates in this release

Description

Workload partitioning pins OpenShift Container Platform and Day 2 Operator pods that are part of the DU profile to the reserved CPU set and removes the reserved CPU from node accounting. This leaves all unreserved CPU cores available for user workloads. This leaves all non-reserved CPU cores available for user workloads. Workload partitioning is enabled through a capability set in installation parameters: cpuPartitioningMode: AllNodes. The set of management partition cores are set with the reserved CPU set that you configure in the PerformanceProfile CR.

Limits and requirements

  • Namespace and Pod CRs must be annotated to allow the pod to be applied to the management partition

  • Pods with CPU limits cannot be allocated to the partition. This is because mutation can change the pod QoS.

  • For more information about the minimum number of CPUs that can be allocated to the management partition, see "Node Tuning Operator".

Engineering considerations

  • Workload partitioning pins all management pods to reserved cores. A sufficient number of cores must be allocated to the reserved set to account for operating system, management pods, and expected spikes in CPU use that occur when the workload starts, the node reboots, or other system events happen.

Additional resources

Cluster tuning

New in this release

  • No reference design updates in this release

Description

See "Cluster capabilities" for a full list of components that can be disabled by using the cluster capabilities feature.

Limits and requirements

  • Cluster capabilities are not available for installer-provisioned installation methods.

Engineering considerations

  • In clusters running OpenShift Container Platform 4.16 and later, the cluster does not automatically revert to cgroup v1 when a PerformanceProfile is applied. If workloads running on the cluster require cgroup v1, the cluster must be configured for cgroup v1. For more information, see "Enabling Linux control group version 1 (cgroup v1)". You should make this configuration as part of the initial cluster deployment.

    Support for cgroup v1 is planned for removal in OpenShift Container Platform 4.19. Clusters running cgroup v1 must transition to cgroup v2.

The following table lists the required platform tuning configurations:

Table 2. Cluster capabilities configurations
Feature Description

Remove optional cluster capabilities

Reduce the OpenShift Container Platform footprint by disabling optional cluster Operators on single-node OpenShift clusters only.

  • Remove all optional Operators except the Node Tuning Operator, Operator Lifecycle Manager, and the Ingress Operator.

Configure cluster monitoring

Configure the monitoring stack for reduced footprint by doing the following:

  • Disable the local alertmanager and telemeter components.

  • If you use RHACM observability, the CR must be augmented with appropriate additionalAlertManagerConfigs CRs to forward alerts to the hub cluster.

  • Reduce the Prometheus retention period to 24h.

    The RHACM hub cluster aggregates managed cluster metrics.

Disable networking diagnostics

Disable networking diagnostics for single-node OpenShift because they are not required.

Configure a single OperatorHub catalog source

Configure the cluster to use a single catalog source that contains only the Operators required for a RAN DU deployment. Each catalog source increases the CPU use on the cluster. Using a single CatalogSource fits within the platform CPU budget.

Disable the Console Operator

If the cluster was deployed with the console disabled, the Console CR (ConsoleOperatorDisable.yaml) is not needed. If the cluster was deployed with the console enabled, you must apply the Console CR.
Additional resources

Machine configuration

New in this release

  • No reference design updates in this release

Limits and requirements

The CRI-O wipe disable MachineConfig CR assumes that images on disk are static other than during scheduled maintenance in defined maintenance windows. To ensure the images are static, do not set the pod imagePullPolicy field to Always.

Table 3. Machine configuration options
Feature Description

Container Runtime

Sets the container runtime to crun for all node roles.

Kubelet config and container mount namespace hiding

Reduces the frequency of kubelet housekeeping and eviction monitoring, which reduces CPU usage

SCTP

Optional configuration (enabled by default)

Kdump

Optional configuration (enabled by default) Enables kdump to capture debug information when a kernel panic occurs. The reference CRs that enable kdump have an increased memory reservation based on the set of drivers and kernel modules included in the reference configuration.

CRI-O wipe disable

Disables automatic wiping of the CRI-O image cache after unclean shutdown

SR-IOV-related kernel arguments

Include additional SR-IOV-related arguments in the kernel command line

Set RCU Normal

Systemd service that sets rcu_normal after the system finishes startup

One-shot time sync

Runs a one-time NTP system time synchronization job for control plane or worker nodes.
Additional resources

Telco RAN DU deployment components

The following sections describe the various OpenShift Container Platform components and configurations that you use to configure the hub cluster with RHACM.

Red Hat Advanced Cluster Management

New in this release

  • No reference design updates in this release

Description

RHACM provides Multi Cluster Engine (MCE) installation and ongoing lifecycle management functionality for deployed clusters. You manage cluster configuration and upgrades declaratively by applying Policy custom resources (CRs) to clusters during maintenance windows.

RHACM provides the following functionality:

  • Zero touch provisioning (ZTP) of clusters using the MCE component in RHACM.

  • Configuration, upgrades, and cluster status through the RHACM policy controller.

  • During managed cluster installation, RHACM can apply labels to individual nodes as configured through the ClusterInstance CR.

Limits and requirements

  • A single hub cluster supports up to 3500 deployed single-node OpenShift clusters with 5 Policy CRs bound to each cluster.

Engineering considerations

  • Use RHACM policy hub-side templating to better scale cluster configuration. You can significantly reduce the number of policies by using a single group policy or small number of general group policies where the group and per-cluster values are substituted into templates.

  • Cluster specific configuration: managed clusters typically have some number of configuration values that are specific to the individual cluster. These configurations should be managed using RHACM policy hub-side templating with values pulled from ConfigMap CRs based on the cluster name.

  • To save CPU resources on managed clusters, policies that apply static configurations should be unbound from managed clusters after GitOps ZTP installation of the cluster.

Additional resources

SiteConfig Operator

New in this release

  • No RDS updates in this release

Description

The SiteConfig Operator is a template-driven solution designed to provision clusters through various installation methods. It introduces the unified ClusterInstance API, which replaces the deprecated SiteConfig API. By leveraging the ClusterInstance API, the SiteConfig Operator improves cluster provisioning by providing the following:

  • Better isolation of definitions from installation methods

  • Unification of Git and non-Git workflows

  • Consistent APIs across installation methods

  • Enhanced scalability

  • Increased flexibility with custom installation templates

  • Valuable insights for troubleshooting deployment issues

The SiteConfig Operator provides validated default installation templates to facilitate cluster deployment through both the Assisted Installer and Image-based Installer provisioning methods:

  • Assisted Installer automates the deployment of OpenShift Container Platform clusters by leveraging predefined configurations and validated host setups. It ensures that the target infrastructure meets OpenShift Container Platform requirements. The Assisted Installer streamlines the installation process while minimizing time and complexity compared to manual setup.

  • Image-based Installer expedites the deployment of single-node OpenShift clusters by utilizing preconfigured and validated OpenShift Container Platform seed images. Seed images are preinstalled on target hosts, enabling rapid reconfiguration and deployment. The Image-based Installer is particularly well-suited for remote or disconnected environments, because it simplifies the cluster creation process and significantly reduces deployment time.

Limits and requirements

  • A single hub cluster supports up to 3500 deployed single-node OpenShift clusters.

Topology Aware Lifecycle Manager

New in this release

  • No reference design updates in this release

Description

Topology Aware Lifecycle Manager is an Operator that runs only on the hub cluster for managing how changes like cluster upgrades, Operator upgrades, and cluster configuration are rolled out to the network. TALM supports the following features:

  • Progressive rollout of policy updates to fleets of clusters in user configurable batches.

  • Per-cluster actions add ztp-done labels or other user-configurable labels following configuration changes to managed clusters.

  • Precaching of single-node OpenShift clusters images: TALM supports optional pre-caching of OpenShift, OLM Operator, and additional user images to single-node OpenShift clusters before initiating an upgrade. The precaching feature is not applicable when using the recommended image-based upgrade method for upgrading single-node OpenShift clusters.

    • Specifying optional pre-caching configurations with PreCachingConfig CRs. Review the sample reference PreCachingConfig CR for more information.

    • Excluding unused images with configurable filtering.

    • Enabling before and after pre-caching storage space validations with configurable space-required parameters.

Limits and requirements

  • Supports concurrent cluster deployment in batches of 400

  • Pre-caching and backup are limited to single-node OpenShift clusters only

Engineering considerations

  • The PreCachingConfig CR is optional and does not need to be created if you only need to precache platform-related OpenShift and OLM Operator images.

  • The PreCachingConfig CR must be applied before referencing it in the ClusterGroupUpgrade CR.

  • Only policies with the ran.openshift.io/ztp-deploy-wave annotation are automatically applied by TALM during cluster installation.

  • Any policy can be remediated by TALM under control of a user created ClusterGroupUpgrade CR.

Additional resources

GitOps Operator and GitOps ZTP

New in this release

  • No reference design updates in this release

Description

GitOps Operator and GitOps ZTP provide a GitOps-based infrastructure for managing cluster deployment and configuration. Cluster definitions and configurations are maintained as a declarative state in Git. You can apply ClusterInstance CRs to the hub cluster where the SiteConfig Operator renders them as installation CRs. In earlier releases, a GitOps ZTP plugin supported the generation of installation CRs from SiteConfig CRs. This plugin is now deprecated. A separate GitOps ZTP plugin is available to enable automatic wrapping of configuration CRs into policies based on the PolicyGenerator or PolicyGenTemplate CR.

You can deploy and manage multiple versions of OpenShift Container Platform on managed clusters by using the baseline reference configuration CRs. You can use custom CRs alongside the baseline CRs. To maintain multiple per-version policies simultaneously, use Git to manage the versions of the source and policy CRs by using PolicyGenerator or PolicyGenTemplate CRs.

Limits and requirements

  • 300 ClusterInstance CRs per ArgoCD application. Multiple applications can be used to achieve the maximum number of clusters supported by a single hub cluster

  • Content in the source-crs/ directory in Git overrides content provided in the ZTP plugin container, as Git takes precedence in the search path.

  • The source-crs/ directory is specifically expected to be located in the same directory as the kustomization.yaml file, which includes PolicyGenerator or PolicyGenTemplate CRs as a generator. Alternative locations for the source-crs/ directory are not supported in this context.

Engineering considerations

  • For multi-node cluster upgrades, you can pause MachineConfigPool (MCP) CRs during maintenance windows by setting the paused field to true. You can increase the number of simultaneously updated nodes per MCP CR by configuring the maxUnavailable setting in the MCP CR. The MaxUnavailable field defines the percentage of nodes in the pool that can be simultaneously unavailable during a MachineConfig update. Set maxUnavailable to the maximum tolerable value. This reduces the number of reboots in a cluster during upgrades which results in shorter upgrade times. When you finally unpause the MCP CR, all the changed configurations are applied with a single reboot.

  • During cluster installation, you can pause custom MCP CRs by setting the paused field to true and setting maxUnavailable to 100% to improve installation times.

  • Keep reference CRs and custom CRs under different directories. Doing this allows you to patch and update the reference CRs by simple replacement of all directory contents without touching the custom CRs. When managing multiple versions, the following best practices are recommended:

    • Keep all source CRs and policy creation CRs in Git repositories to ensure consistent generation of policies for each OpenShift Container Platform version based solely on the contents in Git.

    • Keep reference source CRs in a separate directory from custom CRs. This facilitates easy update of reference CRs as required.

  • To avoid confusion or unintentional overwrites when updating content, it is highly recommended to use unique and distinguishable names for custom CRs in the source-crs/ directory and extra manifests in Git.

  • Extra installation manifests are referenced in the ClusterInstance CR through a ConfigMap CR. The ConfigMap CR should be stored alongside the ClusterInstance CR in Git, serving as the single source of truth for the cluster. If needed, you can use a ConfigMap generator to create the ConfigMap CR.

Additional resources

Agent-based Installer

New in this release

  • No reference design updates in this release

Description

The optional Agent-based Installer component provides installation capabilities without centralized infrastructure. The installation program creates an ISO image that you mount to the server. When the server boots it installs OpenShift Container Platform and supplied extra manifests. The Agent-based Installer allows you to install OpenShift Container Platform without a hub cluster. A container image registry is required for cluster installation.

Limits and requirements

  • You can supply a limited set of additional manifests at installation time.

  • You must include MachineConfiguration CRs that are required by the RAN DU use case.

Engineering considerations

  • The Agent-based Installer provides a baseline OpenShift Container Platform installation.

  • You install Day 2 Operators and the remainder of the RAN DU use case configurations after installation.

Additional resources

Telco RAN DU reference configuration CRs

Use the following custom resources (CRs) to configure and deploy OpenShift Container Platform clusters with the telco RAN DU profile. Use the CRs to form the common baseline used in all the specific use models unless otherwise indicated.

You can extract the complete set of RAN DU CRs from the ztp-site-generate container image. See Preparing the GitOps ZTP site configuration repository for more information.
Additional resources

Cluster tuning reference CRs

Table 4. Cluster tuning CRs
Component Reference CR Description Optional

Cluster capabilities

example-sno.yaml

Representative SiteConfig CR to install single-node OpenShift with the RAN DU profile

No

Console disable

ConsoleOperatorDisable.yaml

Disables the Console Operator.

No

Disconnected registry

09-openshift-marketplace-ns.yaml

Defines a dedicated namespace for managing the OpenShift Operator Marketplace.

No

Disconnected registry

DefaultCatsrc.yaml

Configures the catalog source for the disconnected registry.

No

Disconnected registry

DisableOLMPprof.yaml

Disables performance profiling for OLM.

No

Disconnected registry

DisconnectedICSP.yaml

Configures disconnected registry image content source policy.

No

Disconnected registry

OperatorHub.yaml

Optional, for multi-node clusters only. Configures the OperatorHub in OpenShift, disabling all default Operator sources. Not required for single-node OpenShift installs with marketplace capability disabled.

No

Monitoring configuration

ReduceMonitoringFootprint.yaml

Reduces the monitoring footprint by disabling Alertmanager and Telemeter, and sets Prometheus retention to 24 hours

No

Network diagnostics disable

DisableSnoNetworkDiag.yaml

Configures the cluster network settings to disable built-in network troubleshooting and diagnostic features.

No

Day 2 Operators reference CRs

Table 5. Day 2 Operators CRs
Component Reference CR Description Optional

Cluster Logging Operator

ClusterLogForwarder.yaml

Configures log forwarding for the cluster.

No

Cluster Logging Operator

ClusterLogNS.yaml

Configures the namespace for cluster logging.

No

Cluster Logging Operator

ClusterLogOperGroup.yaml

Configures Operator group for cluster logging.

No

Cluster Logging Operator

ClusterLogServiceAccount.yaml

New in 4.18. Configures the cluster logging service account.

No

Cluster Logging Operator

ClusterLogServiceAccountAuditBinding.yaml

New in 4.18. Configures the cluster logging service account.

No

Cluster Logging Operator

ClusterLogServiceAccountInfrastructureBinding.yaml

New in 4.18. Configures the cluster logging service account.

No

Cluster Logging Operator

ClusterLogSubscription.yaml

Manages installation and updates for the Cluster Logging Operator.

No

Lifecycle Agent

ImageBasedUpgrade.yaml

Manage the image-based upgrade process in OpenShift.

Yes

Lifecycle Agent

LcaSubscription.yaml

Manages installation and updates for the LCA Operator.

Yes

Lifecycle Agent

LcaSubscriptionNS.yaml

Configures namespace for LCA subscription.

Yes

Lifecycle Agent

LcaSubscriptionOperGroup.yaml

Configures the Operator group for the LCA subscription.

Yes

Local Storage Operator

StorageClass.yaml

Defines a storage class with a Delete reclaim policy and no dynamic provisioning in the cluster.

No

Local Storage Operator

StorageLV.yaml

Configures local storage devices for the example-storage-class in the openshift-local-storage namespace, specifying device paths and filesystem type.

No

Local Storage Operator

StorageNS.yaml

Creates the namespace with annotations for workload management and the deployment wave for the Local Storage Operator.

No

Local Storage Operator

StorageOperGroup.yaml

Creates the Operator group for the Local Storage Operator.

No

Local Storage Operator

StorageSubscription.yaml

Creates the namespace for the Local Storage Operator with annotations for workload management and deployment wave.

No

LVM Operator

LVMOperatorStatus.yaml

Verifies the installation or upgrade of the LVM Storage Operator.

Yes

LVM Operator

StorageLVMCluster.yaml

Defines an LVM cluster configuration, with placeholders for storage device classes and volume group settings. Optional substitute for the Local Storage Operator.

No

LVM Operator

StorageLVMSubscription.yaml

Manages installation and updates of the LVMS Operator. Optional substitute for the Local Storage Operator.

No

LVM Operator

StorageLVMSubscriptionNS.yaml

Creates the namespace for the LVMS Operator with labels and annotations for cluster monitoring and workload management. Optional substitute for the Local Storage Operator.

No

LVM Operator

StorageLVMSubscriptionOperGroup.yaml

Defines the target namespace for the LVMS Operator. Optional substitute for the Local Storage Operator.

No

Node Tuning Operator

PerformanceProfile.yaml

Configures node performance settings in an OpenShift cluster, optimizing for low latency and real-time workloads.

No

Node Tuning Operator

TunedPerformancePatch.yaml

Applies performance tuning settings, including scheduler groups and service configurations for nodes in the specific namespace.

No

PTP fast event notifications

PtpConfigBoundaryForEvent.yaml

Configures PTP settings for PTP boundary clocks with additional options for event synchronization. Dependent on cluster role.

No

PTP fast event notifications

PtpConfigForHAForEvent.yaml

Configures PTP for highly available boundary clocks with additional PTP fast event settings. Dependent on cluster role.

No

PTP fast event notifications

PtpConfigMasterForEvent.yaml

Configures PTP for PTP grandmaster clocks with additional PTP fast event settings. Dependent on cluster role.

No

PTP fast event notifications

PtpConfigSlaveForEvent.yaml

Configures PTP for PTP ordinary clocks with additional PTP fast event settings. Dependent on cluster role.

No

PTP fast event notifications

PtpOperatorConfigForEvent.yaml

Overrides the default OperatorConfig. Configures the PTP Operator specifying node selection criteria for running PTP daemons in the openshift-ptp namespace.

No

PTP Operator

PtpConfigBoundary.yaml

Configures PTP settings for PTP boundary clocks. Dependent on cluster role.

No

PTP Operator

PtpConfigDualCardGmWpc.yaml

Configures PTP grandmaster clock settings for hosts that have dual NICs. Dependent on cluster role.

No

PTP Operator

PtpConfigGmWpc.yaml

Configures PTP grandmaster clock settings for hosts that have a single NIC. Dependent on cluster role.

No

PTP Operator

PtpConfigSlave.yaml

Configures PTP settings for a PTP ordinary clock. Dependent on cluster role.

No

PTP Operator

PtpOperatorConfig.yaml

Configures the PTP Operator settings, specifying node selection criteria for running PTP daemons in the openshift-ptp namespace.

No

PTP Operator

PtpSubscription.yaml

Manages installation and updates of the PTP Operator in the openshift-ptp namespace.

No

PTP Operator

PtpSubscriptionNS.yaml

Configures the namespace for the PTP Operator.

No

PTP Operator

PtpSubscriptionOperGroup.yaml

Configures the Operator group for the PTP Operator.

No

PTP Operator (high availability)

PtpConfigBoundary.yaml

Configures PTP settings for highly available PTP boundary clocks.

No

PTP Operator (high availability)

PtpConfigForHA.yaml

Configures PTP settings for highly available PTP boundary clocks.

No

SR-IOV FEC Operator

AcceleratorsNS.yaml

Configures namespace for the VRAN Acceleration Operator. Optional part of application workload.

Yes

SR-IOV FEC Operator

AcceleratorsOperGroup.yaml

Configures the Operator group for the VRAN Acceleration Operator. Optional part of application workload.

Yes

SR-IOV FEC Operator

AcceleratorsSubscription.yaml

Manages installation and updates for the VRAN Acceleration Operator. Optional part of application workload.

Yes

SR-IOV FEC Operator

SriovFecClusterConfig.yaml

Configures SR-IOV FPGA Ethernet Controller (FEC) settings for nodes, specifying drivers, VF amount, and node selection.

Yes

SR-IOV Operator

SriovNetwork.yaml

Defines an SR-IOV network configuration, with placeholders for various network settings.

No

SR-IOV Operator

SriovNetworkNodePolicy.yaml

Configures SR-IOV network settings for specific nodes, including device type, RDMA support, physical function names, and the number of virtual functions.

No

SR-IOV Operator

SriovOperatorConfig.yaml

Configures SR-IOV Network Operator settings, including node selection, injector, and webhook options.

No

SR-IOV Operator

SriovOperatorConfigForSNO.yaml

Configures the SR-IOV Network Operator settings for single-node OpenShift, including node selection, injector, webhook options, and disabling node drain, in the openshift-sriov-network-operator namespace.

No

SR-IOV Operator

SriovSubscription.yaml

Manages the installation and updates of the SR-IOV Network Operator.

No

SR-IOV Operator

SriovSubscriptionNS.yaml

Creates the namespace for the SR-IOV Network Operator with specific annotations for workload management and deployment waves.

No

SR-IOV Operator

SriovSubscriptionOperGroup.yaml

Defines the target namespace for the SR-IOV Network Operators, enabling their management and deployment within this namespace.

No

Machine configuration reference CRs

Table 6. Machine configuration CRs
Component Reference CR Description Optional

Container runtime (crun)

enable-crun-master.yaml

Configures the container runtime (crun) for control plane nodes.

No

Container runtime (crun)

enable-crun-worker.yaml

Configures the container runtime (crun) for worker nodes.

No

CRI-O wipe disable

99-crio-disable-wipe-master.yaml

Disables automatic CRI-O cache wipe following a reboot for on control plane nodes.

No

CRI-O wipe disable

99-crio-disable-wipe-worker.yaml

Disables automatic CRI-O cache wipe following a reboot for on worker nodes.

No

Kdump enable

06-kdump-master.yaml

Configures kdump crash reporting on master nodes.

No

Kdump enable

06-kdump-worker.yaml

Configures kdump crash reporting on worker nodes.

No

Kubelet configuration and container mount hiding

01-container-mount-ns-and-kubelet-conf-master.yaml

Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on control plane nodes.

No

Kubelet configuration and container mount hiding

01-container-mount-ns-and-kubelet-conf-worker.yaml

Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on worker nodes.

No

One-shot time sync

99-sync-time-once-master.yaml

Synchronizes time once on master nodes.

No

One-shot time sync

99-sync-time-once-worker.yaml

Synchronizes time once on worker nodes.

No

SCTP

03-sctp-machine-config-master.yaml

Loads the SCTP kernel module on master nodes.

Yes

SCTP

03-sctp-machine-config-worker.yaml

Loads the SCTP kernel module on worker nodes.

Yes

Set RCU normal

08-set-rcu-normal-master.yaml

Disables rcu_expedited by setting rcu_normal after the control plane node has booted.

No

Set RCU normal

08-set-rcu-normal-worker.yaml

Disables rcu_expedited by setting rcu_normal after the worker node has booted.

No

SRIOV-related kernel arguments

07-sriov-related-kernel-args-master.yaml

Enables SR-IOV support on master nodes.

No

Comparing a cluster with the telco RAN DU reference configuration

After you deploy a telco RAN DU cluster, you can use the cluster-compare plugin to assess the cluster’s compliance with the telco RAN DU reference design specifications (RDS). The cluster-compare plugin is an OpenShift CLI (oc) plugin. The plugin uses a telco RAN DU reference configuration to validate the cluster with the telco RAN DU custom resources (CRs).

The plugin-specific reference configuration for telco RAN DU is packaged in a container image with the telco RAN DU CRs.

For further information about the cluster-compare plugin, see "Understanding the cluster-compare plugin".

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have credentials to access the registry.redhat.io container image registry.

  • You installed the cluster-compare plugin.

Procedure

  1. Login to the container image registry with your credentials by running the following command:

    $ podman login registry.redhat.io

  2. Extract the content from the ztp-site-generate-rhel8 container image by running the following commands::

    $ podman pull registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.18
    $ mkdir -p ./out
    $ podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.18 extract /home/ztp --tar | tar x -C ./out

  3. Compare the configuration for your cluster to the reference configuration by running the following command:

    $ oc cluster-compare -r out/reference/metadata.yaml
    Example output
    ...

**********************************

Cluster CR: config.openshift.io/v1_OperatorHub_cluster (1)
Reference File: required/other/operator-hub.yaml (2)
Diff Output: diff -u -N /tmp/MERGED-2801470219/config-openshift-io-v1_operatorhub_cluster /tmp/LIVE-2569768241/config-openshift-io-v1_operatorhub_cluster
--- /tmp/MERGED-2801470219/config-openshift-io-v1_operatorhub_cluster	2024-12-12 14:13:22.898756462 +0000
+++ /tmp/LIVE-2569768241/config-openshift-io-v1_operatorhub_cluster	2024-12-12 14:13:22.898756462 +0000
@@ -1,6 +1,6 @@
 apiVersion: config.openshift.io/v1
 kind: OperatorHub
 metadata:
+  annotations: (3)
+    include.release.openshift.io/hypershift: "true"
   name: cluster
-spec:
-  disableAllDefaultSources: true

**********************************

Summary (4)
CRs with diffs: 11/12 (5)
CRs in reference missing from the cluster: 40 (6)
optional-image-registry:
  image-registry:
    Missing CRs: (7)
    - optional/image-registry/ImageRegistryPV.yaml
optional-ptp-config:
  ptp-config:
    One of the following is required:
    - optional/ptp-config/PtpConfigBoundary.yaml
    - optional/ptp-config/PtpConfigGmWpc.yaml
    - optional/ptp-config/PtpConfigDualCardGmWpc.yaml
    - optional/ptp-config/PtpConfigForHA.yaml
    - optional/ptp-config/PtpConfigMaster.yaml
    - optional/ptp-config/PtpConfigSlave.yaml
    - optional/ptp-config/PtpConfigSlaveForEvent.yaml
    - optional/ptp-config/PtpConfigForHAForEvent.yaml
    - optional/ptp-config/PtpConfigMasterForEvent.yaml
    - optional/ptp-config/PtpConfigBoundaryForEvent.yaml
  ptp-operator-config:
    One of the following is required:
    - optional/ptp-config/PtpOperatorConfig.yaml
    - optional/ptp-config/PtpOperatorConfigForEvent.yaml
optional-storage:
  storage:
    Missing CRs:
    - optional/local-storage-operator/StorageLV.yaml

...

No CRs are unmatched to reference CRs (8)
Metadata Hash: 09650c31212be9a44b99315ec14d2e7715ee194a5d68fb6d24f65fd5ddbe3c3c (9)
No patched CRs (10)
    1 The CR under comparison. The plugin displays each CR with a difference from the corresponding template.
    2 The template matching with the CR for comparison.
    3 The output in Linux diff format shows the difference between the template and the cluster CR.
    4 After the plugin reports the line diffs for each CR, the summary of differences are reported.
    5 The number of CRs in the comparison with differences from the corresponding templates.
    6 The number of CRs represented in the reference configuration, but missing from the live cluster.
    7 The list of CRs represented in the reference configuration, but missing from the live cluster.
    8 The CRs that did not match to a corresponding template in the reference configuration.
    9 The metadata hash identifies the reference configuration.
    10 The list of patched CRs.

Telco RAN DU 4.18 validated software components

The Red Hat telco RAN DU 4.18 solution has been validated using the following Red Hat software products for OpenShift Container Platform managed clusters.

Table 7. Telco RAN DU managed cluster validated software components
Component Software version

Managed cluster version

4.18

Cluster Logging Operator

6.11

Local Storage Operator

4.18

OpenShift API for Data Protection (OADP)

1.4

PTP Operator

4.18

SR-IOV Operator

4.18

SRIOV-FEC Operator

2.10

Lifecycle Agent

4.18

[1] This table will be updated when the aligned Cluster Logging Operator version 6.2 is released.

Telco RAN DU 4.18 hub cluster validated software components

The Red Hat telco RAN 4.18 solution has been validated using the following Red Hat software products for OpenShift Container Platform hub clusters.

Table 8. Telco hub cluster validated software components
Component Software version

Hub cluster version

4.18

Red Hat Advanced Cluster Management (RHACM)

2.121

Red Hat OpenShift GitOps

1.14

GitOps ZTP site generate plugins

4.18

Topology Aware Lifecycle Manager (TALM)

4.18

[1] This table will be updated when the aligned RHACM version 2.13 is released.