×

After installing OpenShift Container Platform, you can further expand and customize your cluster to your requirements through certain node tasks.

Adding RHEL compute machines to an OpenShift Container Platform cluster

Understand and work with RHEL compute nodes.

About adding RHEL compute nodes to a cluster

In OpenShift Container Platform 4.10, you have the option of using Red Hat Enterprise Linux (RHEL) machines as compute machines in your cluster if you use a user-provisioned infrastructure installation on the x86_64 architecture. You must use Red Hat Enterprise Linux CoreOS (RHCOS) machines for the control plane machines in your cluster.

If you choose to use RHEL compute machines in your cluster, you are responsible for all operating system life cycle management and maintenance. You must perform system updates, apply patches, and complete all other required tasks.

  • Because removing OpenShift Container Platform from a machine in the cluster requires destroying the operating system, you must use dedicated hardware for any RHEL machines that you add to the cluster.

  • Swap memory is disabled on all RHEL machines that you add to your OpenShift Container Platform cluster. You cannot enable swap memory on these machines.

You must add any RHEL compute machines to the cluster after you initialize the control plane.

System requirements for RHEL compute nodes

The Red Hat Enterprise Linux (RHEL) compute machine hosts in your OpenShift Container Platform environment must meet the following minimum hardware specifications and system-level requirements:

  • You must have an active OpenShift Container Platform subscription on your Red Hat account. If you do not, contact your sales representative for more information.

  • Production environments must provide compute machines to support your expected workloads. As a cluster administrator, you must calculate the expected workload and add about 10% for overhead. For production environments, allocate enough resources so that a node host failure does not affect your maximum capacity.

  • Each system must meet the following hardware requirements:

    • Physical or virtual system, or an instance running on a public or private IaaS.

    • Base OS: RHEL 8.4 or 8.5 with "Minimal" installation option.

      Adding RHEL 7 compute machines to an OpenShift Container Platform cluster is not supported.

      If you have RHEL 7 compute machines that were previously supported in a past OpenShift Container Platform version, you cannot upgrade them to RHEL 8. You must deploy new RHEL 8 hosts, and the old RHEL 7 hosts should be removed. See the "Deleting nodes" section for more information.

      For the most recent list of major functionality that has been deprecated or removed within OpenShift Container Platform, refer to the Deprecated and removed features section of the OpenShift Container Platform release notes.

    • If you deployed OpenShift Container Platform in FIPS mode, you must enable FIPS on the RHEL machine before you boot it. See Installing a RHEL 8 system with FIPS mode enabled in the RHEL 8 documentation.

The use of FIPS Validated / Modules in Process cryptographic libraries is only supported on OpenShift Container Platform deployments on the x86_64 architecture.

  • NetworkManager 1.0 or later.

  • 1 vCPU.

  • Minimum 8 GB RAM.

  • Minimum 15 GB hard disk space for the file system containing /var/.

  • Minimum 1 GB hard disk space for the file system containing /usr/local/bin/.

  • Minimum 1 GB hard disk space for the file system containing its temporary directory. The temporary system directory is determined according to the rules defined in the tempfile module in the Python standard library.

    • Each system must meet any additional requirements for your system provider. For example, if you installed your cluster on VMware vSphere, your disks must be configured according to its storage guidelines and the disk.enableUUID=true attribute must be set.

    • Each system must be able to access the cluster’s API endpoints by using DNS-resolvable hostnames. Any network security access control that is in place must allow system access to the cluster’s API service endpoints.

Additional resources

Certificate signing requests management

Because your cluster has limited access to automatic machine management when you use infrastructure that you provision, you must provide a mechanism for approving cluster certificate signing requests (CSRs) after installation. The kube-controller-manager only approves the kubelet client CSRs. The machine-approver cannot guarantee the validity of a serving certificate that is requested by using kubelet credentials because it cannot confirm that the correct machine issued the request. You must determine and implement a method of verifying the validity of the kubelet serving certificate requests and approving them.

Preparing the machine to run the playbook

Before you can add compute machines that use Red Hat Enterprise Linux (RHEL) as the operating system to an OpenShift Container Platform 4.10 cluster, you must prepare a RHEL 7 or 8 machine to run an Ansible playbook that adds the new node to the cluster. This machine is not part of the cluster but must be able to access it.

Prerequisites
  • Install the OpenShift CLI (oc) on the machine that you run the playbook on.

  • Log in as a user with cluster-admin permission.

Procedure
  1. Ensure that the kubeconfig file for the cluster and the installation program that you used to install the cluster are on the RHEL machine. One way to accomplish this is to use the same machine that you used to install the cluster.

  2. Configure the machine to access all of the RHEL hosts that you plan to use as compute machines. You can use any method that your company allows, including a bastion with an SSH proxy or a VPN.

  3. Configure a user on the machine that you run the playbook on that has SSH access to all of the RHEL hosts.

    If you use SSH key-based authentication, you must manage the key with an SSH agent.

  4. If you have not already done so, register the machine with RHSM and attach a pool with an OpenShift subscription to it:

    1. Register the machine with RHSM:

      # subscription-manager register --username=<user_name> --password=<password>
    2. Pull the latest subscription data from RHSM:

      # subscription-manager refresh
    3. List the available subscriptions:

      # subscription-manager list --available --matches '*OpenShift*'
    4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

      # subscription-manager attach --pool=<pool_id>
  5. Enable the repositories required by OpenShift Container Platform 4.10:

    • On a RHEL 8 system, run the following command:

      # subscription-manager repos \
          --enable="rhel-8-for-x86_64-baseos-rpms" \
          --enable="rhel-8-for-x86_64-appstream-rpms" \
          --enable="ansible-2.9-for-rhel-8-x86_64-rpms" \
          --enable="rhocp-4.10-for-rhel-8-x86_64-rpms"
    • On a RHEL 7 system, run the following command:

      # subscription-manager repos \
          --enable="rhel-7-server-rpms" \
          --enable="rhel-7-server-extras-rpms" \
          --enable="rhel-7-server-ansible-2.9-rpms" \
          --enable="rhel-7-server-ose-4.10-rpms"

      As of OpenShift Container Platform 4.10.23, running the Ansible playbooks on RHEL 7 is deprecated, and suggested only for the purpose of updating existing installations. Starting with OpenShift Container Platform 4.11, the Ansible playbooks are provided only for RHEL 8.

  6. Install the required packages, including openshift-ansible:

    # yum install openshift-ansible openshift-clients jq

    The openshift-ansible package provides installation program utilities and pulls in other packages that you require to add a RHEL compute node to your cluster, such as Ansible, playbooks, and related configuration files. The openshift-clients provides the oc CLI, and the jq package improves the display of JSON output on your command line.

Preparing a RHEL compute node

Before you add a Red Hat Enterprise Linux (RHEL) machine to your OpenShift Container Platform cluster, you must register each host with Red Hat Subscription Manager (RHSM), attach an active OpenShift Container Platform subscription, and enable the required repositories.

  1. On each host, register with RHSM:

    # subscription-manager register --username=<user_name> --password=<password>
  2. Pull the latest subscription data from RHSM:

    # subscription-manager refresh
  3. List the available subscriptions:

    # subscription-manager list --available --matches '*OpenShift*'
  4. In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:

    # subscription-manager attach --pool=<pool_id>
  5. Disable all yum repositories:

    1. Disable all the enabled RHSM repositories:

      # subscription-manager repos --disable="*"
    2. List the remaining yum repositories and note their names under repo id, if any:

      # yum repolist
    3. Use yum-config-manager to disable the remaining yum repositories:

      # yum-config-manager --disable <repo_id>

      Alternatively, disable all repositories:

      # yum-config-manager --disable \*

      Note that this might take a few minutes if you have a large number of available repositories

  6. Enable only the repositories required by OpenShift Container Platform 4.10:

    # subscription-manager repos \
        --enable="rhel-8-for-x86_64-baseos-rpms" \
        --enable="rhel-8-for-x86_64-appstream-rpms" \
        --enable="rhocp-4.10-for-rhel-8-x86_64-rpms" \
        --enable="fast-datapath-for-rhel-8-x86_64-rpms"
  7. Stop and disable firewalld on the host:

    # systemctl disable --now firewalld.service

    You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker.

Adding a RHEL compute machine to your cluster

You can add compute machines that use Red Hat Enterprise Linux as the operating system to an OpenShift Container Platform 4.10 cluster.

Prerequisites
  • You installed the required packages and performed the necessary configuration on the machine that you run the playbook on.

  • You prepared the RHEL hosts for installation.

Procedure

Perform the following steps on the machine that you prepared to run the playbook:

  1. Create an Ansible inventory file that is named /<path>/inventory/hosts that defines your compute machine hosts and required variables:

    [all:vars]
    ansible_user=root (1)
    #ansible_become=True (2)
    
    openshift_kubeconfig_path="~/.kube/config" (3)
    
    [new_workers] (4)
    mycluster-rhel8-0.example.com
    mycluster-rhel8-1.example.com
    1 Specify the user name that runs the Ansible tasks on the remote compute machines.
    2 If you do not specify root for the ansible_user, you must set ansible_become to True and assign the user sudo permissions.
    3 Specify the path and file name of the kubeconfig file for your cluster.
    4 List each RHEL machine to add to your cluster. You must provide the fully-qualified domain name for each host. This name is the hostname that the cluster uses to access the machine, so set the correct public or private name to access the machine.
  2. Navigate to the Ansible playbook directory:

    $ cd /usr/share/ansible/openshift-ansible
  3. Run the playbook:

    $ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml (1)
    1 For <path>, specify the path to the Ansible inventory file that you created.

Required parameters for the Ansible hosts file

You must define the following parameters in the Ansible hosts file before you add Red Hat Enterprise Linux (RHEL) compute machines to your cluster.

Parameter Description Values

ansible_user

The SSH user that allows SSH-based authentication without requiring a password. If you use SSH key-based authentication, then you must manage the key with an SSH agent.

A user name on the system. The default value is root.

ansible_become

If the values of ansible_user is not root, you must set ansible_become to True, and the user that you specify as the ansible_user must be configured for passwordless sudo access.

True. If the value is not True, do not specify and define this parameter.

openshift_kubeconfig_path

Specifies a path and file name to a local directory that contains the kubeconfig file for your cluster.

The path and name of the configuration file.

Optional: Removing RHCOS compute machines from a cluster

After you add the Red Hat Enterprise Linux (RHEL) compute machines to your cluster, you can optionally remove the Red Hat Enterprise Linux CoreOS (RHCOS) compute machines to free up resources.

Prerequisites
  • You have added RHEL compute machines to your cluster.

Procedure
  1. View the list of machines and record the node names of the RHCOS compute machines:

    $ oc get nodes -o wide
  2. For each RHCOS compute machine, delete the node:

    1. Mark the node as unschedulable by running the oc adm cordon command:

      $ oc adm cordon <node_name> (1)
      1 Specify the node name of one of the RHCOS compute machines.
    2. Drain all the pods from the node:

      $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets (1)
      1 Specify the node name of the RHCOS compute machine that you isolated.
    3. Delete the node:

      $ oc delete nodes <node_name> (1)
      1 Specify the node name of the RHCOS compute machine that you drained.
  3. Review the list of compute machines to ensure that only the RHEL nodes remain:

    $ oc get nodes -o wide
  4. Remove the RHCOS machines from the load balancer for your cluster’s compute machines. You can delete the virtual machines or reimage the physical hardware for the RHCOS compute machines.

Adding RHCOS compute machines to an OpenShift Container Platform cluster

You can add more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines to your OpenShift Container Platform cluster on bare metal.

Before you add more compute machines to a cluster that you installed on bare metal infrastructure, you must create RHCOS machines for it to use. You can either use an ISO image or network PXE booting to create the machines.

Prerequisites

  • You installed a cluster on bare metal.

  • You have installation media and Red Hat Enterprise Linux CoreOS (RHCOS) images that you used to create your cluster. If you do not have these files, you must obtain them by following the instructions in the installation procedure.

Creating more RHCOS machines using an ISO image

You can create more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines for your bare metal cluster by using an ISO image to create the machines.

Prerequisites
  • Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.

Procedure
  1. Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:

    • Burn the ISO image to a disk and boot it directly.

    • Use ISO redirection with a LOM interface.

  2. After the instance boots, press the TAB or E key to edit the kernel command line.

  3. Add the parameters to the kernel command line:

    coreos.inst.install_dev=sda (1)
    coreos.inst.ignition_url=http://example.com/worker.ign (2)
    
    1 Specify the block device of the system to install to.
    2 Specify the URL of the compute Ignition config file. Only HTTP and HTTPS protocols are supported.
  4. Press Enter to complete the installation. After RHCOS installs, the system reboots. After the system reboots, it applies the Ignition config file that you specified.

  5. Continue to create more compute machines for your cluster.

Creating more RHCOS machines by PXE or iPXE booting

You can create more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines for your bare metal cluster by using PXE or iPXE booting.

Prerequisites
  • Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.

  • Obtain the URLs of the RHCOS ISO image, compressed metal BIOS, kernel, and initramfs files that you uploaded to your HTTP server during cluster installation.

  • You have access to the PXE booting infrastructure that you used to create the machines for your OpenShift Container Platform cluster during installation. The machines must boot from their local disks after RHCOS is installed on them.

  • If you use UEFI, you have access to the grub.conf file that you modified during OpenShift Container Platform installation.

Procedure
  1. Confirm that your PXE or iPXE installation for the RHCOS images is correct.

    • For PXE:

      DEFAULT pxeboot
      TIMEOUT 20
      PROMPT 0
      LABEL pxeboot
          KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> (1)
          APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img (2)
      1 Specify the location of the live kernel file that you uploaded to your HTTP server.
      2 Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.

This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the APPEND line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

  • For iPXE:

    kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img (1)
    initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img (2)
    1 Specify locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS.
    2 Specify the location of the initramfs file that you uploaded to your HTTP server.

This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console= arguments to the kernel line. For example, add console=tty0 console=ttyS0 to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.

  1. Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.

Approving the certificate signing requests for your machines

When you add machines to a cluster, two pending certificate signing requests (CSRs) are generated for each machine that you added. You must confirm that these CSRs are approved or, if necessary, approve them yourself. The client requests must be approved first, followed by the server requests.

Prerequisites
  • You added machines to your cluster.

Procedure
  1. Confirm that the cluster recognizes the machines:

    $ oc get nodes
    Example output
    NAME      STATUS    ROLES   AGE  VERSION
    master-0  Ready     master  63m  v1.23.0
    master-1  Ready     master  63m  v1.23.0
    master-2  Ready     master  64m  v1.23.0

    The output lists all of the machines that you created.

    The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

  2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

    $ oc get csr
    Example output
    NAME        AGE     REQUESTOR                                                                   CONDITION
    csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
    csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
    ...

    In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

  3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

    Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

    For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

    • To approve them individually, run the following command for each valid CSR:

      $ oc adm certificate approve <csr_name> (1)
      1 <csr_name> is the name of a CSR from the list of current CSRs.
    • To approve all pending CSRs, run the following command:

      $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

      Some Operators might not become available until some CSRs are approved.

  4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

    $ oc get csr
    Example output
    NAME        AGE     REQUESTOR                                                                   CONDITION
    csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
    csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
    ...
  5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

    • To approve them individually, run the following command for each valid CSR:

      $ oc adm certificate approve <csr_name> (1)
      1 <csr_name> is the name of a CSR from the list of current CSRs.
    • To approve all pending CSRs, run the following command:

      $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
  6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

    $ oc get nodes
    Example output
    NAME      STATUS    ROLES   AGE  VERSION
    master-0  Ready     master  73m  v1.23.0
    master-1  Ready     master  73m  v1.23.0
    master-2  Ready     master  74m  v1.23.0
    worker-0  Ready     worker  11m  v1.23.0
    worker-1  Ready     worker  11m  v1.23.0

    It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Additional information

Adding a new RHCOS worker node with a custom /var partition in AWS

OpenShift Container Platform supports partitioning devices during installation by using machine configs that are processed during the bootstrap. However, if you use /var partitioning, the device name must be determined at installation and cannot be changed. You cannot add different instance types as nodes if they have a different device naming schema. For example, if you configured the /var partition with the default AWS device name for m4.large instances, dev/xvdb, you cannot directly add an AWS m5.large instance, as m5.large instances use a /dev/nvme1n1 device by default. The device might fail to partition due to the different naming schema.

The procedure in this section shows how to add a new Red Hat Enterprise Linux CoreOS (RHCOS) compute node with an instance that uses a different device name from what was configured at installation. You create a custom user data secret and configure a new machine set. These steps are specific to an AWS cluster. The principles apply to other cloud deployments also. However, the device naming schema is different for other deployments and should be determined on a per-case basis.

Procedure
  1. On a command line, change to the openshift-machine-api namespace:

    $ oc project openshift-machine-api
  2. Create a new secret from the worker-user-data secret:

    1. Export the userData section of the secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.userData | base64decode}}' | jq > userData.txt
    2. Edit the text file to add the storage, filesystems, and systemd stanzas for the partitions you want to use for the new node. You can specify any Ignition configuration parameters as needed.

      Do not change the values in the ignition stanza.

      {
        "ignition": {
          "config": {
            "merge": [
              {
                "source": "https:...."
              }
            ]
          },
          "security": {
            "tls": {
              "certificateAuthorities": [
                {
                  "source": "data:text/plain;charset=utf-8;base64,.....=="
                }
              ]
            }
          },
          "version": "3.2.0"
        },
        "storage": {
          "disks": [
            {
              "device": "/dev/nvme1n1", (1)
              "partitions": [
                {
                  "label": "var",
                  "sizeMiB": 50000, (2)
                  "startMiB": 0 (3)
                }
              ]
            }
          ],
          "filesystems": [
            {
              "device": "/dev/disk/by-partlabel/var", (4)
              "format": "xfs", (5)
              "path": "/var" (6)
            }
          ]
        },
        "systemd": {
          "units": [ (7)
            {
              "contents": "[Unit]\nBefore=local-fs.target\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nOptions=defaults,pquota\n[Install]\nWantedBy=local-fs.target\n",
              "enabled": true,
              "name": "var.mount"
            }
          ]
        }
      }
      1 Specifies an absolute path to the AWS block device.
      2 Specifies the size of the data partition in Mebibytes.
      3 Specifies the start of the partition in Mebibytes. When adding a data partition to the boot disk, a minimum value of 25000 MB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of RHCOS might overwrite the beginning of the data partition.
      4 Specifies an absolute path to the /var partition.
      5 Specifies the filesystem format.
      6 Specifies the mount-point of the filesystem while Ignition is running relative to where the root filesystem will be mounted. This is not necessarily the same as where it should be mounted in the real root, but it is encouraged to make it the same.
      7 Defines a systemd mount unit that mounts the /dev/disk/by-partlabel/var device to the /var partition.
    3. Extract the disableTemplating section from the work-user-data secret to a text file:

      $ oc get secret worker-user-data --template='{{index .data.disableTemplating | base64decode}}' | jq > disableTemplating.txt
    4. Create the new user data secret file from the two text files. This user data secret passes the additional node partition information in the userData.txt file to the newly created node.

      $ oc create secret generic worker-user-data-x5 --from-file=userData=userData.txt --from-file=disableTemplating=disableTemplating.txt
  3. Create a new machine set for the new node:

    1. Create a new machine set YAML file, similar to the following, which is configured for AWS. Add the required partitions and the newly-created user data secret:

      Use an existing machine set as a template and change the parameters as needed for the new node.

      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: auto-52-92tf4
        name: worker-us-east-2-nvme1n1 (1)
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: auto-52-92tf4
            machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: auto-52-92tf4
              machine.openshift.io/cluster-api-machine-role: worker
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
          spec:
            metadata: {}
            providerSpec:
              value:
                ami:
                  id: ami-0c2dbd95931a
                apiVersion: awsproviderconfig.openshift.io/v1beta1
                blockDevices:
                - DeviceName: /dev/nvme1n1 (2)
                  ebs:
                    encrypted: true
                    iops: 0
                    volumeSize: 120
                    volumeType: gp2
                - DeviceName: /dev/nvme1n2 (3)
                  ebs:
                    encrypted: true
                    iops: 0
                    volumeSize: 50
                    volumeType: gp2
                credentialsSecret:
                  name: aws-cloud-credentials
                deviceIndex: 0
                iamInstanceProfile:
                  id: auto-52-92tf4-worker-profile
                instanceType: m6i.large
                kind: AWSMachineProviderConfig
                metadata:
                  creationTimestamp: null
                placement:
                  availabilityZone: us-east-2b
                  region: us-east-2
                securityGroups:
                - filters:
                  - name: tag:Name
                    values:
                    - auto-52-92tf4-worker-sg
                subnet:
                  id: subnet-07a90e5db1
                tags:
                - name: kubernetes.io/cluster/auto-52-92tf4
                  value: owned
                userDataSecret:
                  name: worker-user-data-x5 (4)
      
      1 Specifies a name for the new node.
      2 Specifies an absolute path to the AWS block device, here an encrypted EBS volume.
      3 Optional. Specifies an additional EBS volume.
      4 Specifies the user data secret file.
    2. Create the machine set:

      $ oc create -f <file-name>.yaml

      The machines might take a few moments to become available.

  4. Verify that the new partition and nodes are created:

    1. Verify that the machine set is created:

      $ oc get machineset
      Example output
      NAME                                               DESIRED   CURRENT   READY   AVAILABLE   AGE
      ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1a        1         1         1       1           124m
      ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1b        2         2         2       2           124m
      worker-us-east-2-nvme1n1                           1         1         1       1           2m35s (1)
      
      1 This is the new machine set.
    2. Verify that the new node is created:

      $ oc get nodes
      Example output
      NAME                           STATUS   ROLES    AGE     VERSION
      ip-10-0-128-78.ec2.internal    Ready    worker   117m    v1.23.0+60f5a1c
      ip-10-0-146-113.ec2.internal   Ready    master   127m    v1.23.0+60f5a1c
      ip-10-0-153-35.ec2.internal    Ready    worker   118m    v1.23.0+60f5a1c
      ip-10-0-176-58.ec2.internal    Ready    master   126m    v1.23.0+60f5a1c
      ip-10-0-217-135.ec2.internal   Ready    worker   2m57s   v1.23.0+60f5a1c (1)
      ip-10-0-225-248.ec2.internal   Ready    master   127m    v1.23.0+60f5a1c
      ip-10-0-245-59.ec2.internal    Ready    worker   116m    v1.23.0+60f5a1c
      1 This is new new node.
    3. Verify that the custom /var partition is created on the new node:

      $ oc debug node/<node-name> -- chroot /host lsblk

      For example:

      $ oc debug node/ip-10-0-217-135.ec2.internal -- chroot /host lsblk
      Example output
      NAME        MAJ:MIN  RM  SIZE RO TYPE MOUNTPOINT
      nvme0n1     202:0    0   120G  0 disk
      |-nvme0n1p1 202:1    0     1M  0 part
      |-nvme0n1p2 202:2    0   127M  0 part
      |-nvme0n1p3 202:3    0   384M  0 part /boot
      `-nvme0n1p4 202:4    0 119.5G  0 part /sysroot
      nvme1n1     202:16   0    50G  0 disk
      `-nvme1n1p1 202:17   0  48.8G  0 part /var (1)
      
      1 The nvme1n1 device is mounted to the /var partition.
Additional resources
  • For more information on how OpenShift Container Platform uses disk partitioning, see Disk partitioning.

Deploying machine health checks

Understand and deploy machine health checks.

This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational.

About machine health checks

Machine health checks automatically repair unhealthy machines in a particular machine pool.

To monitor machine health, create a resource to define the configuration for a controller. Set a condition to check, such as staying in the NotReady status for five minutes or displaying a permanent condition in the node-problem-detector, and a label for the set of machines to monitor.

You cannot apply a machine health check to a machine with the master role.

The controller that observes a MachineHealthCheck resource checks for the defined condition. If a machine fails the health check, the machine is automatically deleted and one is created to take its place. When a machine is deleted, you see a machine deleted event.

To limit disruptive impact of the machine deletion, the controller drains and deletes only one node at a time. If there are more unhealthy machines than the maxUnhealthy threshold allows for in the targeted pool of machines, remediation stops and therefore enables manual intervention.

Consider the timeouts carefully, accounting for workloads and requirements.

  • Long timeouts can result in long periods of downtime for the workload on the unhealthy machine.

  • Too short timeouts can result in a remediation loop. For example, the timeout for checking the NotReady status must be long enough to allow the machine to complete the startup process.

To stop the check, remove the resource.

Limitations when deploying machine health checks

There are limitations to consider before deploying a machine health check:

  • Only machines owned by a machine set are remediated by a machine health check.

  • Control plane machines are not currently supported and are not remediated if they are unhealthy.

  • If the node for a machine is removed from the cluster, a machine health check considers the machine to be unhealthy and remediates it immediately.

  • If the corresponding node for a machine does not join the cluster after the nodeStartupTimeout, the machine is remediated.

  • A machine is remediated immediately if the Machine resource phase is Failed.

Sample MachineHealthCheck resource

The MachineHealthCheck resource for all cloud-based installation types, and other than bare metal, resembles the following YAML file:

apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
  name: example (1)
  namespace: openshift-machine-api
spec: