# subscription-manager register --username=<user_name> --password=<password>
After installing OpenShift Container Platform, you can further expand and customize your cluster to your requirements through certain node tasks.
Understand and work with RHEL compute nodes.
In OpenShift Container Platform 4.10, you have the option of using Red Hat Enterprise Linux (RHEL) machines as compute machines in your cluster if you use a user-provisioned infrastructure installation on the x86_64
architecture. You must use Red Hat Enterprise Linux CoreOS (RHCOS) machines for the control plane machines in your cluster.
If you choose to use RHEL compute machines in your cluster, you are responsible for all operating system life cycle management and maintenance. You must perform system updates, apply patches, and complete all other required tasks.
|
You must add any RHEL compute machines to the cluster after you initialize the control plane.
The Red Hat Enterprise Linux (RHEL) compute machine hosts in your OpenShift Container Platform environment must meet the following minimum hardware specifications and system-level requirements:
You must have an active OpenShift Container Platform subscription on your Red Hat account. If you do not, contact your sales representative for more information.
Production environments must provide compute machines to support your expected workloads. As a cluster administrator, you must calculate the expected workload and add about 10% for overhead. For production environments, allocate enough resources so that a node host failure does not affect your maximum capacity.
Each system must meet the following hardware requirements:
Physical or virtual system, or an instance running on a public or private IaaS.
Base OS: RHEL 8.4 or 8.5 with "Minimal" installation option.
Adding RHEL 7 compute machines to an OpenShift Container Platform cluster is not supported. If you have RHEL 7 compute machines that were previously supported in a past OpenShift Container Platform version, you cannot upgrade them to RHEL 8. You must deploy new RHEL 8 hosts, and the old RHEL 7 hosts should be removed. See the "Deleting nodes" section for more information. For the most recent list of major functionality that has been deprecated or removed within OpenShift Container Platform, refer to the Deprecated and removed features section of the OpenShift Container Platform release notes. |
If you deployed OpenShift Container Platform in FIPS mode, you must enable FIPS on the RHEL machine before you boot it. See Installing a RHEL 8 system with FIPS mode enabled in the RHEL 8 documentation.
The use of FIPS Validated / Modules in Process cryptographic libraries is only supported on OpenShift Container Platform deployments on the |
NetworkManager 1.0 or later.
1 vCPU.
Minimum 8 GB RAM.
Minimum 15 GB hard disk space for the file system containing /var/
.
Minimum 1 GB hard disk space for the file system containing /usr/local/bin/
.
Minimum 1 GB hard disk space for the file system containing its temporary directory. The temporary system directory is determined according to the rules defined in the tempfile module in the Python standard library.
Each system must meet any additional requirements for your system provider. For example, if you installed your cluster on VMware vSphere, your disks must be configured according to its storage guidelines and the disk.enableUUID=true
attribute must be set.
Each system must be able to access the cluster’s API endpoints by using DNS-resolvable hostnames. Any network security access control that is in place must allow system access to the cluster’s API service endpoints.
Because your cluster has limited access to automatic machine management when you use infrastructure that you provision, you must provide a mechanism for approving cluster certificate signing requests (CSRs) after installation. The kube-controller-manager
only approves the kubelet client CSRs. The machine-approver
cannot guarantee the validity of a serving certificate that is requested by using kubelet credentials because it cannot confirm that the correct machine issued the request. You must determine and implement a method of verifying the validity of the kubelet serving certificate requests and approving them.
Before you can add compute machines that use Red Hat Enterprise Linux (RHEL) as the operating system to an OpenShift Container Platform 4.10 cluster, you must prepare a RHEL 7 or 8 machine to run an Ansible playbook that adds the new node to the cluster. This machine is not part of the cluster but must be able to access it.
Install the OpenShift CLI (oc
) on the machine that you run the playbook on.
Log in as a user with cluster-admin
permission.
Ensure that the kubeconfig
file for the cluster and the installation program that you used to install the cluster are on the RHEL machine. One way to accomplish this is to use the same machine that you used to install the cluster.
Configure the machine to access all of the RHEL hosts that you plan to use as compute machines. You can use any method that your company allows, including a bastion with an SSH proxy or a VPN.
Configure a user on the machine that you run the playbook on that has SSH access to all of the RHEL hosts.
If you use SSH key-based authentication, you must manage the key with an SSH agent. |
If you have not already done so, register the machine with RHSM and attach a pool with an OpenShift
subscription to it:
Register the machine with RHSM:
# subscription-manager register --username=<user_name> --password=<password>
Pull the latest subscription data from RHSM:
# subscription-manager refresh
List the available subscriptions:
# subscription-manager list --available --matches '*OpenShift*'
In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:
# subscription-manager attach --pool=<pool_id>
Enable the repositories required by OpenShift Container Platform 4.10:
On a RHEL 8 system, run the following command:
# subscription-manager repos \
--enable="rhel-8-for-x86_64-baseos-rpms" \
--enable="rhel-8-for-x86_64-appstream-rpms" \
--enable="ansible-2.9-for-rhel-8-x86_64-rpms" \
--enable="rhocp-4.10-for-rhel-8-x86_64-rpms"
On a RHEL 7 system, run the following command:
# subscription-manager repos \
--enable="rhel-7-server-rpms" \
--enable="rhel-7-server-extras-rpms" \
--enable="rhel-7-server-ansible-2.9-rpms" \
--enable="rhel-7-server-ose-4.10-rpms"
As of OpenShift Container Platform 4.10.23, running the Ansible playbooks on RHEL 7 is deprecated, and suggested only for the purpose of updating existing installations. Starting with OpenShift Container Platform 4.11, the Ansible playbooks are provided only for RHEL 8. |
Install the required packages, including openshift-ansible
:
# yum install openshift-ansible openshift-clients jq
The openshift-ansible
package provides installation program utilities and pulls in other packages that you require to add a RHEL compute node to your cluster, such as Ansible, playbooks, and related configuration files. The openshift-clients
provides the oc
CLI, and the jq
package improves the display of JSON output on your command line.
Before you add a Red Hat Enterprise Linux (RHEL) machine to your OpenShift Container Platform cluster, you must register each host with Red Hat Subscription Manager (RHSM), attach an active OpenShift Container Platform subscription, and enable the required repositories.
On each host, register with RHSM:
# subscription-manager register --username=<user_name> --password=<password>
Pull the latest subscription data from RHSM:
# subscription-manager refresh
List the available subscriptions:
# subscription-manager list --available --matches '*OpenShift*'
In the output for the previous command, find the pool ID for an OpenShift Container Platform subscription and attach it:
# subscription-manager attach --pool=<pool_id>
Disable all yum repositories:
Disable all the enabled RHSM repositories:
# subscription-manager repos --disable="*"
List the remaining yum repositories and note their names under repo id
, if any:
# yum repolist
Use yum-config-manager
to disable the remaining yum repositories:
# yum-config-manager --disable <repo_id>
Alternatively, disable all repositories:
# yum-config-manager --disable \*
Note that this might take a few minutes if you have a large number of available repositories
Enable only the repositories required by OpenShift Container Platform 4.10:
# subscription-manager repos \
--enable="rhel-8-for-x86_64-baseos-rpms" \
--enable="rhel-8-for-x86_64-appstream-rpms" \
--enable="rhocp-4.10-for-rhel-8-x86_64-rpms" \
--enable="fast-datapath-for-rhel-8-x86_64-rpms"
Stop and disable firewalld on the host:
# systemctl disable --now firewalld.service
You must not enable firewalld later. If you do, you cannot access OpenShift Container Platform logs on the worker. |
You can add compute machines that use Red Hat Enterprise Linux as the operating system to an OpenShift Container Platform 4.10 cluster.
You installed the required packages and performed the necessary configuration on the machine that you run the playbook on.
You prepared the RHEL hosts for installation.
Perform the following steps on the machine that you prepared to run the playbook:
Create an Ansible inventory file that is named /<path>/inventory/hosts
that defines your compute machine hosts and required variables:
[all:vars] ansible_user=root (1) #ansible_become=True (2) openshift_kubeconfig_path="~/.kube/config" (3) [new_workers] (4) mycluster-rhel8-0.example.com mycluster-rhel8-1.example.com
1 | Specify the user name that runs the Ansible tasks on the remote compute machines. |
2 | If you do not specify root for the ansible_user , you must set ansible_become to True and assign the user sudo permissions. |
3 | Specify the path and file name of the kubeconfig file for your cluster. |
4 | List each RHEL machine to add to your cluster. You must provide the fully-qualified domain name for each host. This name is the hostname that the cluster uses to access the machine, so set the correct public or private name to access the machine. |
Navigate to the Ansible playbook directory:
$ cd /usr/share/ansible/openshift-ansible
Run the playbook:
$ ansible-playbook -i /<path>/inventory/hosts playbooks/scaleup.yml (1)
1 | For <path> , specify the path to the Ansible inventory file that you created. |
You must define the following parameters in the Ansible hosts file before you add Red Hat Enterprise Linux (RHEL) compute machines to your cluster.
Parameter | Description | Values |
---|---|---|
|
The SSH user that allows SSH-based authentication without requiring a password. If you use SSH key-based authentication, then you must manage the key with an SSH agent. |
A user name on the system. The default value is |
|
If the values of |
|
|
Specifies a path and file name to a local directory that contains the |
The path and name of the configuration file. |
After you add the Red Hat Enterprise Linux (RHEL) compute machines to your cluster, you can optionally remove the Red Hat Enterprise Linux CoreOS (RHCOS) compute machines to free up resources.
You have added RHEL compute machines to your cluster.
View the list of machines and record the node names of the RHCOS compute machines:
$ oc get nodes -o wide
For each RHCOS compute machine, delete the node:
Mark the node as unschedulable by running the oc adm cordon
command:
$ oc adm cordon <node_name> (1)
1 | Specify the node name of one of the RHCOS compute machines. |
Drain all the pods from the node:
$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets (1)
1 | Specify the node name of the RHCOS compute machine that you isolated. |
Delete the node:
$ oc delete nodes <node_name> (1)
1 | Specify the node name of the RHCOS compute machine that you drained. |
Review the list of compute machines to ensure that only the RHEL nodes remain:
$ oc get nodes -o wide
Remove the RHCOS machines from the load balancer for your cluster’s compute machines. You can delete the virtual machines or reimage the physical hardware for the RHCOS compute machines.
You can add more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines to your OpenShift Container Platform cluster on bare metal.
Before you add more compute machines to a cluster that you installed on bare metal infrastructure, you must create RHCOS machines for it to use. You can either use an ISO image or network PXE booting to create the machines.
You installed a cluster on bare metal.
You have installation media and Red Hat Enterprise Linux CoreOS (RHCOS) images that you used to create your cluster. If you do not have these files, you must obtain them by following the instructions in the installation procedure.
You can create more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines for your bare metal cluster by using an ISO image to create the machines.
Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.
Use the ISO file to install RHCOS on more compute machines. Use the same method that you used when you created machines before you installed the cluster:
Burn the ISO image to a disk and boot it directly.
Use ISO redirection with a LOM interface.
After the instance boots, press the TAB
or E
key to edit the kernel command line.
Add the parameters to the kernel command line:
coreos.inst.install_dev=sda (1)
coreos.inst.ignition_url=http://example.com/worker.ign (2)
1 | Specify the block device of the system to install to. |
2 | Specify the URL of the compute Ignition config file. Only HTTP and HTTPS protocols are supported. |
Press Enter
to complete the installation. After RHCOS installs, the system reboots. After the system reboots, it applies the Ignition config file that you specified.
Continue to create more compute machines for your cluster.
You can create more Red Hat Enterprise Linux CoreOS (RHCOS) compute machines for your bare metal cluster by using PXE or iPXE booting.
Obtain the URL of the Ignition config file for the compute machines for your cluster. You uploaded this file to your HTTP server during installation.
Obtain the URLs of the RHCOS ISO image, compressed metal BIOS, kernel
, and initramfs
files that you uploaded to your HTTP server during cluster installation.
You have access to the PXE booting infrastructure that you used to create the machines for your OpenShift Container Platform cluster during installation. The machines must boot from their local disks after RHCOS is installed on them.
If you use UEFI, you have access to the grub.conf
file that you modified during OpenShift Container Platform installation.
Confirm that your PXE or iPXE installation for the RHCOS images is correct.
For PXE:
DEFAULT pxeboot TIMEOUT 20 PROMPT 0 LABEL pxeboot KERNEL http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> (1) APPEND initrd=http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img (2)
1 | Specify the location of the live kernel file that you uploaded to your HTTP server. |
2 | Specify locations of the RHCOS files that you uploaded to your HTTP server. The initrd parameter value is the location of the live initramfs file, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS. |
This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console=
arguments to the APPEND
line. For example, add console=tty0 console=ttyS0
to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.
For iPXE:
kernel http://<HTTP_server>/rhcos-<version>-live-kernel-<architecture> initrd=main coreos.inst.install_dev=/dev/sda coreos.inst.ignition_url=http://<HTTP_server>/worker.ign coreos.live.rootfs_url=http://<HTTP_server>/rhcos-<version>-live-rootfs.<architecture>.img (1) initrd --name main http://<HTTP_server>/rhcos-<version>-live-initramfs.<architecture>.img (2)
1 | Specify locations of the RHCOS files that you uploaded to your HTTP server. The kernel parameter value is the location of the kernel file, the initrd=main argument is needed for booting on UEFI systems, the coreos.inst.ignition_url parameter value is the location of the worker Ignition config file, and the coreos.live.rootfs_url parameter value is the location of the live rootfs file. The coreos.inst.ignition_url and coreos.live.rootfs_url parameters only support HTTP and HTTPS. |
2 | Specify the location of the initramfs file that you uploaded to your HTTP server. |
This configuration does not enable serial console access on machines with a graphical console. To configure a different console, add one or more console=
arguments to the kernel
line. For example, add console=tty0 console=ttyS0
to set the first PC serial port as the primary console and the graphical console as a secondary console. For more information, see How does one set up a serial terminal and/or console in Red Hat Enterprise Linux?.
Use the PXE or iPXE infrastructure to create the required compute machines for your cluster.
When you add machines to a cluster, two pending certificate signing requests (CSRs) are generated for each machine that you added. You must confirm that these CSRs are approved or, if necessary, approve them yourself. The client requests must be approved first, followed by the server requests.
You added machines to your cluster.
Confirm that the cluster recognizes the machines:
$ oc get nodes
NAME STATUS ROLES AGE VERSION
master-0 Ready master 63m v1.23.0
master-1 Ready master 63m v1.23.0
master-2 Ready master 64m v1.23.0
The output lists all of the machines that you created.
The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved. |
Review the pending CSRs and ensure that you see the client requests with the Pending
or Approved
status for each machine that you added to the cluster:
$ oc get csr
NAME AGE REQUESTOR CONDITION
csr-8b2br 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
csr-8vnps 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
...
In this example, two machines are joining the cluster. You might see more approved CSRs in the list.
If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending
status, approve the CSRs for your cluster machines:
Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the |
For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the |
To approve them individually, run the following command for each valid CSR:
$ oc adm certificate approve <csr_name> (1)
1 | <csr_name> is the name of a CSR from the list of current CSRs. |
To approve all pending CSRs, run the following command:
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve
Some Operators might not become available until some CSRs are approved. |
Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:
$ oc get csr
NAME AGE REQUESTOR CONDITION
csr-bfd72 5m26s system:node:ip-10-0-50-126.us-east-2.compute.internal Pending
csr-c57lv 5m26s system:node:ip-10-0-95-157.us-east-2.compute.internal Pending
...
If the remaining CSRs are not approved, and are in the Pending
status, approve the CSRs for your cluster machines:
To approve them individually, run the following command for each valid CSR:
$ oc adm certificate approve <csr_name> (1)
1 | <csr_name> is the name of a CSR from the list of current CSRs. |
To approve all pending CSRs, run the following command:
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
After all client and server CSRs have been approved, the machines have the Ready
status. Verify this by running the following command:
$ oc get nodes
NAME STATUS ROLES AGE VERSION
master-0 Ready master 73m v1.23.0
master-1 Ready master 73m v1.23.0
master-2 Ready master 74m v1.23.0
worker-0 Ready worker 11m v1.23.0
worker-1 Ready worker 11m v1.23.0
It can take a few minutes after approval of the server CSRs for the machines to transition to the |
For more information on CSRs, see Certificate Signing Requests.
/var
partition in AWSOpenShift Container Platform supports partitioning devices during installation by using machine configs that are processed during the bootstrap. However, if you use /var
partitioning, the device name must be determined at installation and cannot be changed. You cannot add different instance types as nodes if they have a different device naming schema. For example, if you configured the /var
partition with the default AWS device name for m4.large
instances, dev/xvdb
, you cannot directly add an AWS m5.large
instance, as m5.large
instances use a /dev/nvme1n1
device by default. The device might fail to partition due to the different naming schema.
The procedure in this section shows how to add a new Red Hat Enterprise Linux CoreOS (RHCOS) compute node with an instance that uses a different device name from what was configured at installation. You create a custom user data secret and configure a new machine set. These steps are specific to an AWS cluster. The principles apply to other cloud deployments also. However, the device naming schema is different for other deployments and should be determined on a per-case basis.
On a command line, change to the openshift-machine-api
namespace:
$ oc project openshift-machine-api
Create a new secret from the worker-user-data
secret:
Export the userData
section of the secret to a text file:
$ oc get secret worker-user-data --template='{{index .data.userData | base64decode}}' | jq > userData.txt
Edit the text file to add the storage
, filesystems
, and systemd
stanzas for the partitions you want to use for the new node. You can specify any Ignition configuration parameters as needed.
Do not change the values in the |
{
"ignition": {
"config": {
"merge": [
{
"source": "https:...."
}
]
},
"security": {
"tls": {
"certificateAuthorities": [
{
"source": "data:text/plain;charset=utf-8;base64,.....=="
}
]
}
},
"version": "3.2.0"
},
"storage": {
"disks": [
{
"device": "/dev/nvme1n1", (1)
"partitions": [
{
"label": "var",
"sizeMiB": 50000, (2)
"startMiB": 0 (3)
}
]
}
],
"filesystems": [
{
"device": "/dev/disk/by-partlabel/var", (4)
"format": "xfs", (5)
"path": "/var" (6)
}
]
},
"systemd": {
"units": [ (7)
{
"contents": "[Unit]\nBefore=local-fs.target\n[Mount]\nWhere=/var\nWhat=/dev/disk/by-partlabel/var\nOptions=defaults,pquota\n[Install]\nWantedBy=local-fs.target\n",
"enabled": true,
"name": "var.mount"
}
]
}
}
1 | Specifies an absolute path to the AWS block device. |
2 | Specifies the size of the data partition in Mebibytes. |
3 | Specifies the start of the partition in Mebibytes. When adding a data partition to the boot disk, a minimum value of 25000 MB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of RHCOS might overwrite the beginning of the data partition. |
4 | Specifies an absolute path to the /var partition. |
5 | Specifies the filesystem format. |
6 | Specifies the mount-point of the filesystem while Ignition is running relative to where the root filesystem will be mounted. This is not necessarily the same as where it should be mounted in the real root, but it is encouraged to make it the same. |
7 | Defines a systemd mount unit that mounts the /dev/disk/by-partlabel/var device to the /var partition. |
Extract the disableTemplating
section from the work-user-data
secret to a text file:
$ oc get secret worker-user-data --template='{{index .data.disableTemplating | base64decode}}' | jq > disableTemplating.txt
Create the new user data secret file from the two text files. This user data secret passes the additional node partition information in the userData.txt
file to the newly created node.
$ oc create secret generic worker-user-data-x5 --from-file=userData=userData.txt --from-file=disableTemplating=disableTemplating.txt
Create a new machine set for the new node:
Create a new machine set YAML file, similar to the following, which is configured for AWS. Add the required partitions and the newly-created user data secret:
Use an existing machine set as a template and change the parameters as needed for the new node. |
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
labels:
machine.openshift.io/cluster-api-cluster: auto-52-92tf4
name: worker-us-east-2-nvme1n1 (1)
namespace: openshift-machine-api
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: auto-52-92tf4
machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: auto-52-92tf4
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: auto-52-92tf4-worker-us-east-2b
spec:
metadata: {}
providerSpec:
value:
ami:
id: ami-0c2dbd95931a
apiVersion: awsproviderconfig.openshift.io/v1beta1
blockDevices:
- DeviceName: /dev/nvme1n1 (2)
ebs:
encrypted: true
iops: 0
volumeSize: 120
volumeType: gp2
- DeviceName: /dev/nvme1n2 (3)
ebs:
encrypted: true
iops: 0
volumeSize: 50
volumeType: gp2
credentialsSecret:
name: aws-cloud-credentials
deviceIndex: 0
iamInstanceProfile:
id: auto-52-92tf4-worker-profile
instanceType: m6i.large
kind: AWSMachineProviderConfig
metadata:
creationTimestamp: null
placement:
availabilityZone: us-east-2b
region: us-east-2
securityGroups:
- filters:
- name: tag:Name
values:
- auto-52-92tf4-worker-sg
subnet:
id: subnet-07a90e5db1
tags:
- name: kubernetes.io/cluster/auto-52-92tf4
value: owned
userDataSecret:
name: worker-user-data-x5 (4)
1 | Specifies a name for the new node. |
2 | Specifies an absolute path to the AWS block device, here an encrypted EBS volume. |
3 | Optional. Specifies an additional EBS volume. |
4 | Specifies the user data secret file. |
Create the machine set:
$ oc create -f <file-name>.yaml
The machines might take a few moments to become available.
Verify that the new partition and nodes are created:
Verify that the machine set is created:
$ oc get machineset
NAME DESIRED CURRENT READY AVAILABLE AGE
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1a 1 1 1 1 124m
ci-ln-2675bt2-76ef8-bdgsc-worker-us-east-1b 2 2 2 2 124m
worker-us-east-2-nvme1n1 1 1 1 1 2m35s (1)
1 | This is the new machine set. |
Verify that the new node is created:
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-128-78.ec2.internal Ready worker 117m v1.23.0+60f5a1c
ip-10-0-146-113.ec2.internal Ready master 127m v1.23.0+60f5a1c
ip-10-0-153-35.ec2.internal Ready worker 118m v1.23.0+60f5a1c
ip-10-0-176-58.ec2.internal Ready master 126m v1.23.0+60f5a1c
ip-10-0-217-135.ec2.internal Ready worker 2m57s v1.23.0+60f5a1c (1)
ip-10-0-225-248.ec2.internal Ready master 127m v1.23.0+60f5a1c
ip-10-0-245-59.ec2.internal Ready worker 116m v1.23.0+60f5a1c
1 | This is new new node. |
Verify that the custom /var
partition is created on the new node:
$ oc debug node/<node-name> -- chroot /host lsblk
For example:
$ oc debug node/ip-10-0-217-135.ec2.internal -- chroot /host lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 202:0 0 120G 0 disk
|-nvme0n1p1 202:1 0 1M 0 part
|-nvme0n1p2 202:2 0 127M 0 part
|-nvme0n1p3 202:3 0 384M 0 part /boot
`-nvme0n1p4 202:4 0 119.5G 0 part /sysroot
nvme1n1 202:16 0 50G 0 disk
`-nvme1n1p1 202:17 0 48.8G 0 part /var (1)
1 | The nvme1n1 device is mounted to the /var partition. |
For more information on how OpenShift Container Platform uses disk partitioning, see Disk partitioning.
Understand and deploy machine health checks.
This process is not applicable for clusters with manually provisioned machines. You can use the advanced machine management and scaling capabilities only in clusters where the Machine API is operational. |
Machine health checks automatically repair unhealthy machines in a particular machine pool.
To monitor machine health, create a resource to define the configuration for a controller. Set a condition to check, such as staying in the NotReady
status for five minutes or displaying a permanent condition in the node-problem-detector, and a label for the set of machines to monitor.
You cannot apply a machine health check to a machine with the master role. |
The controller that observes a MachineHealthCheck
resource checks for the defined condition. If a machine fails the health check, the machine is automatically deleted and one is created to take its place. When a machine is deleted, you see a machine deleted
event.
To limit disruptive impact of the machine deletion, the controller drains and deletes only one node at a time. If there are more unhealthy machines than the maxUnhealthy
threshold allows for in the targeted pool of machines, remediation stops and therefore enables manual intervention.
Consider the timeouts carefully, accounting for workloads and requirements.
|
To stop the check, remove the resource.
There are limitations to consider before deploying a machine health check:
Only machines owned by a machine set are remediated by a machine health check.
Control plane machines are not currently supported and are not remediated if they are unhealthy.
If the node for a machine is removed from the cluster, a machine health check considers the machine to be unhealthy and remediates it immediately.
If the corresponding node for a machine does not join the cluster after the nodeStartupTimeout
, the machine is remediated.
A machine is remediated immediately if the Machine
resource phase is Failed
.
The MachineHealthCheck
resource for all cloud-based installation types, and other than bare metal, resembles the following YAML file:
apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
name: example (1)
namespace: openshift-machine-api
spec: