$ oc adm cordon <node1>
As an administrator, you can perform several tasks to make your clusters more efficient.
Evacuating pods allows you to migrate all or selected pods from a given node or nodes.
You can only evacuate pods backed by a replication controller. The replication controller creates new pods on other nodes and removes the existing pods from the specified node(s).
Bare pods, meaning those not backed by a replication controller, are unaffected by default. You can evacuate a subset of pods by specifying a pod-selector. Pod selectors are based on labels, so all the pods with the specified label will be evacuated.
Mark the nodes unschedulable before performing the pod evacuation.
Mark the node as unschedulable:
$ oc adm cordon <node1>
node/<node1> cordoned
Check that the node status is Ready,SchedulingDisabled
:
$ oc get node <node1>
NAME STATUS ROLES AGE VERSION
<node1> Ready,SchedulingDisabled worker 1d v1.25.0
Evacuate the pods using one of the following methods:
Evacuate all or selected pods on one or more nodes:
$ oc adm drain <node1> <node2> [--pod-selector=<pod_selector>]
Force the deletion of bare pods using the --force
option. When set to
true
, deletion continues even if there are pods not managed by a replication
controller, replica set, job, daemon set, or stateful set:
$ oc adm drain <node1> <node2> --force=true
Set a period of time in seconds for each pod to
terminate gracefully, use --grace-period
. If negative, the default value specified in the pod will
be used:
$ oc adm drain <node1> <node2> --grace-period=-1
Ignore pods managed by daemon sets using the --ignore-daemonsets
flag set to true
:
$ oc adm drain <node1> <node2> --ignore-daemonsets=true
Set the length of time to wait before giving up using the --timeout
flag. A
value of 0
sets an infinite length of time:
$ oc adm drain <node1> <node2> --timeout=5s
Delete pods even if there are pods using emptyDir
volumes by setting the --delete-emptydir-data
flag to true
. Local data is deleted when the node
is drained:
$ oc adm drain <node1> <node2> --delete-emptydir-data=true
List objects that will be migrated without actually performing the evacuation,
using the --dry-run
option set to true
:
$ oc adm drain <node1> <node2> --dry-run=true
Instead of specifying specific node names (for example, <node1> <node2>
), you
can use the --selector=<node_selector>
option to evacuate pods on selected
nodes.
Mark the node as schedulable when done.
$ oc adm uncordon <node1>
You can update any label on a node.
Node labels are not persisted after a node is deleted even if the node is backed up by a Machine.
Any change to a |
The following command adds or updates labels on a node:
$ oc label node <node> <key_1>=<value_1> ... <key_n>=<value_n>
For example:
$ oc label nodes webconsole-7f7f6 unhealthy=true
You can alternatively apply the following YAML to apply the label:
|
The following command updates all pods in the namespace:
$ oc label pods --all <key_1>=<value_1>
For example:
$ oc label pods --all status=unhealthy
By default, healthy nodes with a Ready
status are
marked as schedulable, which means that you can place new pods on the
node. Manually marking a node as unschedulable blocks any new pods from being
scheduled on the node. Existing pods on the node are not affected.
The following command marks a node or nodes as unschedulable:
$ oc adm cordon <node>
For example:
$ oc adm cordon node1.example.com
node/node1.example.com cordoned
NAME LABELS STATUS
node1.example.com kubernetes.io/hostname=node1.example.com Ready,SchedulingDisabled
The following command marks a currently unschedulable node or nodes as schedulable:
$ oc adm uncordon <node1>
Alternatively, instead of specifying specific node names (for example, <node>
), you can use the --selector=<node_selector>
option to mark selected
nodes as schedulable or unschedulable.
In single-node OpenShift clusters and in OpenShift Container Platform clusters in general, a situation can arise where a node reboot occurs without first draining the node. This can occur where an application pod requesting devices fails with the UnexpectedAdmissionError
error. Deployment
, ReplicaSet
, or DaemonSet
errors are reported because the application pods that require those devices start before the pod serving those devices. You cannot control the order of pod restarts.
While this behavior is to be expected, it can cause a pod to remain on the cluster even though it has failed to deploy successfully. The pod continues to report UnexpectedAdmissionError
. This issue is mitigated by the fact that application pods are typically included in a Deployment
, ReplicaSet
, or DaemonSet
. If a pod is in this error state, it is of little concern because another instance should be running. Belonging to a Deployment
, ReplicaSet
, or DaemonSet
guarantees the successful creation and execution of subsequent pods and ensures the successful deployment of the application.
There is ongoing work upstream to ensure that such pods are gracefully terminated. Until that work is resolved, run the following command for a single-node OpenShift cluster to remove the failed pods:
$ oc delete pods --field-selector status.phase=Failed -n <POD_NAMESPACE>
The option to drain the node is unavailable for single-node OpenShift clusters. |
To delete a node from the OpenShift Container Platform cluster, scale down the appropriate MachineSet
object.
When a cluster is integrated with a cloud provider, you must delete the corresponding machine to delete a node. Do not try to use the |
When you delete a node by using the CLI, the node object is deleted in Kubernetes, but the pods that exist on the node are not deleted. Any bare pods that are not backed by a replication controller become inaccessible to OpenShift Container Platform. Pods backed by replication controllers are rescheduled to other available nodes. You must delete local manifest pods.
If you are running cluster on bare metal, you cannot delete a node by editing |
View the compute machine sets that are in the cluster by running the following command:
$ oc get machinesets -n openshift-machine-api
The compute machine sets are listed in the form of <cluster-id>-worker-<aws-region-az>
.
Scale down the compute machine set by using one of the following methods:
Specify the number of replicas to scale down to by running the following command:
$ oc scale --replicas=2 machineset <machine-set-name> -n openshift-machine-api
Edit the compute machine set custom resource by running the following command:
$ oc edit machineset <machine-set-name> -n openshift-machine-api
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
# ...
name: <machine-set-name>
namespace: openshift-machine-api
# ...
spec:
replicas: 2 (1)
# ...
1 | Specify the number of replicas to scale down to. |
When you delete a node using the CLI, the node object is deleted in Kubernetes, but the pods that exist on the node are not deleted. Any bare pods not backed by a replication controller become inaccessible to OpenShift Container Platform. Pods backed by replication controllers are rescheduled to other available nodes. You must delete local manifest pods.
Delete a node from an OpenShift Container Platform cluster running on bare metal by completing the following steps:
Mark the node as unschedulable:
$ oc adm cordon <node_name>
Drain all pods on the node:
$ oc adm drain <node_name> --force=true
This step might fail if the node is offline or unresponsive. Even if the node does not respond, it might still be running a workload that writes to shared storage. To avoid data corruption, power down the physical hardware before you proceed.
Delete the node from the cluster:
$ oc delete node <node_name>
Although the node object is now deleted from the cluster, it can still rejoin the cluster after reboot or if the kubelet service is restarted. To permanently delete the node and all its data, you must decommission the node.
If you powered down the physical hardware, turn it back on so that the node can rejoin the cluster.