Harden NTP

All nodes in your environment must have accurate and consistent timestamps. Time synchronization spans two distinct domains: the Red Hat OpenShift Container Platform (RHOCP) cluster on the control plane and the RHEL data plane nodes.

Consistent timekeeping is critical for both operational reliability and security:

Security event correlation
Inconsistent timestamps across systems make it impossible to reconstruct the sequence of events during an incident. Log analysis, forensic investigation, and audit trails all depend on synchronized clocks.
Authentication systems
Kerberos-based authentication can refuse to authenticate clients with time irregularities. TLS certificate validation depends on accurate time because a client whose clock is outside the certificate’s validity range will reject the connection.
RHOSO services
High Availability, Red Hat Ceph Storage, and the RHOCP etcd cluster are all affected by time drift. Excessive skew can cause cluster split-brain, leader election failures, or data inconsistency.
Kubernetes lease and certificate expiry
RHOCP uses leases for leader election and cert-manager uses system time for certificate renewal calculations. Clock drift on control plane nodes can cause premature certificate expiry or delayed renewal.

Observe the following NTP security considerations:

Use authenticated NTP
If your internal NTP servers support NTS (Network Time Security) or symmetric key authentication, enable it in chrony.conf to prevent time spoofing attacks. An attacker who can shift a node’s clock can force TLS certificate validation failures or bypass token expiry checks.
Restrict NTP access
Configure your internal NTP servers to accept queries only from your data center networks. Use the chrony allow and deny directives to enforce this.
Monitor time drift
Set up alerting for clock drift exceeding your tolerance threshold. For Kerberos environments, the typical maximum allowable skew is 5 minutes. For TLS validation and HA clusters, even a few seconds of drift can cause problems.
Do not use the hypervisor clock for guest instances
Configure VM instances to synchronize time from your internal NTP servers directly, not from the compute host. The kvm-clock source can drift on heavily loaded hypervisors.

NTP architecture recommendations
Copy link

Network Time Protocol (NTP) is organized in a hierarchical stratum design. Do not connect your data center nodes directly to public stratum 1 or 2 NTP servers. Instead, deploy one or more dedicated NTP servers in your data center for the nodes in your deployment.

Observe the following recommendations:

  • All Red Hat OpenShift Container Platform (RHOCP) cluster nodes must use the same NTP source.
  • All Red Hat OpenStack Services on OpenShift (RHOSO) data plane nodes must use the same NTP source.
  • Both domains should use the same upstream NTP server or pool to guarantee cross-domain consistency.
  • VM instances running on compute nodes should receive time from your dedicated internal NTP servers, not from the hypervisor host.
Important
Clock synchronization between RHOCP nodes and data plane nodes is a documented prerequisite for RHOSO 18.0 deployment. Failure to synchronize clocks across both domains can cause TLS handshake failures, authentication errors, and service instability.

Configure NTP on the control plane
Copy link

Red Hat OpenShift Container Platform (RHOCP) nodes run Red Hat Enterprise Linux CoreOS (RHCOS) that uses chronyd by default. The default configuration points to public *.rhel.pool.ntp.org servers. To point RHOCP nodes at your internal NTP servers, apply a MachineConfig resource.

Procedure
Copy link

  1. Create a chrony.conf file with your internal NTP server and encode it as base64: 
    $ cat > chrony.conf << 'EOF'
server ntp.internal.example.com iburst
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
logdir /var/log/chrony
EOF

$ CHRONY_BASE64=$(base64 -w 0 chrony.conf)
  2. Create MachineConfig resources for both master and worker roles. The following example applies to client nodes. Repeat the step with role: master for master nodes: 
    apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: workers-chrony-configuration
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,<CHRONY_BASE64>
          mode: 0644
          path: /etc/chrony.conf
    • Replace <CHRONY_BASE64> with the output of the base64 encoding: 
      $ oc apply -f workers-chrony-configuration.yaml
$ oc apply -f masters-chrony-configuration.yaml
    Note
    Applying a MachineConfig starts a rolling reboot of the affected nodes. The Machine Config Operator (MCO) drains each node before rebooting. Plan this change during a maintenance window.

Results
Copy link

  • Run the following commands for your RHOCP nodes. Verify that the System clock synchronized field shows yes and that the reference source is your internal NTP server: 
    $ oc debug node/<node-name> -- chroot /host chronyc sources
$ oc debug node/<node-name> -- chroot /host chronyc tracking

Configure NTP on the data plane
Copy link

Data plane nodes with RHEL use chrony to configure NTP. Configure time synchronization with the timesync_ntp_servers Ansible variable in the OpenStackDataPlaneNodeSet CR.

Procedure
Copy link

Add the time synchronization configuration to the ansibleVars section of your OpenStackDataPlaneNodeSet CR: 
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneNodeSet
metadata:
  name: openstack-edpm-compute
spec:
  nodeTemplate:
    ansible:
      ansibleVars:
        timesync_ntp_servers:
          - hostname: ntp.internal.example.com
            iburst: true

You can list multiple NTP servers for redundancy. You can also configure NTP at the individual node level by placing timesync_ntp_servers in the per-node ansibleVars section.

Results
Copy link

  • After deploying or redeploying the data plane, verify chrony on each compute node: 
    $ ssh -i <key> cloud-admin@<compute-ip> chronyc sources
$ ssh -i <key> cloud-admin@<compute-ip> chronyc tracking

RHOSP 17.1 to RHOSO 18.0 NTP comparison
Copy link

Review the NTP differences between Red Hat OpenStack Platform (RHOSP) 17.1 and Red Hat OpenStack Services on OpenShift (RHOSO) 18.0.

Table 1. Comparisons between NTP in RHOSP 17.1 and RHOSO 18.0
RHOSP 17.1 mechanism RHOSO 18.0 equivalent Notes

Chrony on undercloud or overcloud control plane nodes configured by director.

Chrony on RHOCP nodes configured by MachineConfig CR.

Applied by Machine Config Operator and triggers rolling reboot.

Chrony on compute or data plane nodes configured by director heat templates.

timesync_ntp_servers in OpenStackDataPlaneNodeSet CR.

EDPM Ansible configures chrony on RHEL data plane nodes.

Containerized services inherit host time.

Pods inherit RHOCP node time.

Pods use the host kernel clock in both versions.

No undercloud exists.

N/A

Undercloud concept eliminated in RHOSO 18.0.