1. Documentation
    2. history
×

Troubleshooting network issues

How the network interface is selected

For installations on bare metal or with virtual machines that have more than one network interface controller (NIC), the NIC that OpenShift Container Platform uses for communication with the Kubernetes API server is determined by the nodeip-configuration.service service unit that is run by systemd when the node boots. The service iterates through the network interfaces on the node and the first network interface that is configured with a subnet than can host the IP address for the API server is selected for OpenShift Container Platform communication.

After the nodeip-configuration.service service determines the correct NIC, the service creates the /etc/systemd/system/kubelet.service.d/20-nodenet.conf file. The 20-nodenet.conf file sets the KUBELET_NODE_IP environment variable to the IP address that the service selected.

When the kubelet service starts, it reads the value of the environment variable from the 20-nodenet.conf file and sets the IP address as the value to the --node-ip kubelet command-line argument. As a result, the kubelet service uses the selected IP address as the node IP address.

If hardware or networking is reconfigured after installation, it is possible that the nodeip-configuration.service service can select a different NIC after a reboot. In some cases, you might be able to detect that a different NIC is selected by reviewing the INTERNAL-IP column in the output from the oc get nodes -o wide command.

If network communication is disrupted or misconfigured because a different NIC is selected, one strategy for overriding the selection process is to set the correct IP address explicitly. The following list identifies the high-level steps and considerations:

  • Create a shell script that determines the IP address to use for OpenShift Container Platform communication. Have the script create a custom unit file such as /etc/systemd/system/kubelet.service.d/98-nodenet-override.conf. Use the custom unit file, 98-nodenet-override.conf, to set the KUBELET_NODE_IP environment variable to the IP address.

  • Do not overwrite the /etc/systemd/system/kubelet.service.d/20-nodenet.conf file. Specify a file name with a numerically higher value such as 98-nodenet-override.conf in the same directory path. The goal is to have the custom unit file run after 20-nodenet.conf and override the value of the environment variable.

  • Create a machine config object with the shell script as a base64-encoded string and use the Machine Config Operator to deploy the script to the nodes at a file system path such as /usr/local/bin/override-node-ip.sh.

  • Ensure that systemctl daemon-reload runs after the shell script runs. The simplest method is to specify ExecStart=systemctl daemon-reload in the machine config, as shown in the following sample.

Sample machine config to override the network interface for kubelet
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
     machineconfiguration.openshift.io/role: worker
  name: 98-nodenet-override
spec:
  config:
    ignition:
      version: 3.1.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,<encoded_script>
        mode: 0755
        overwrite: true
        path: /usr/local/bin/override-node-ip.sh
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Override node IP detection
          Wants=network-online.target
          Before=kubelet.service
          After=network-online.target
          [Service]
          Type=oneshot
          ExecStart=/usr/local/bin/override-node-ip.sh
          ExecStart=systemctl daemon-reload
          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: nodenet-override.service