A node selector specifies a map of key-value pairs. The rules are defined using custom labels on nodes and selectors specified in pods. You can use specific node selectors to place specific pods on specific nodes, project node selectors to place new pods in a project on specific nodes in that project, or default cluster-wide node selectors to place new pods on specific nodes anywhere in the cluster.

For the pod to be eligible to run on a node, the pod must have the same key-value node selector as the label on the node.

If you are using node selectors and node affinity in the same pod configuration, the following rules control pod placement onto nodes:

  • If you configure both nodeSelector and nodeAffinity, both conditions must be satisfied for the pod to be scheduled onto a candidate node.

  • If you specify multiple nodeSelectorTerms associated with nodeAffinity types, then the pod can be scheduled onto a node if one of the nodeSelectorTerms is satisfied.

  • If you specify multiple matchExpressions associated with nodeSelectorTerms, then the pod can be scheduled onto a node only if all matchExpressions are satisfied.

Using node selectors to control pod placement

You can use node selector labels on pods to control where the pod is scheduled.

With node selectors, OpenShift Container Platform schedules the pods on nodes that contain matching labels.

To add node selectors to an existing pod, add a node selector to the controlling object for that node, such as a ReplicaSet, Daemonset, or StatefulSet. Any existing pods under that controlling object are recreated on a node with a matching label. If you are creating a new pod, you can add the node selector directly to the pod spec.

You can add labels to a node or MachineConfig, but the labels will not persist if the node or machine goes down. Adding the label to the MachineSet ensures that new nodes or machines will have the label.

You cannot add a node selector directly to an existing scheduled pod.

Prerequisite

To add a node selector to existing pods, determine the controlling object for that pod. For example, the router-default-66d5cf9464-m2g75 pod is controlled by the router-default-66d5cf9464 ReplicaSet:

$ oc describe pod router-default-66d5cf9464-7pwkc

Name:               router-default-66d5cf9464-7pwkc
Namespace:          openshift-ingress

....

Controlled By:      ReplicaSet/router-default-66d5cf9464

The web console lists the controlling object under ownerReferences in the pod YAML:

  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: router-default-66d5cf9464
      uid: d81dd094-da26-11e9-a48a-128e7edf0312
      controller: true
      blockOwnerDeletion: true
Procedure
  1. Add labels to a node by using a MachineSet or editing the node directly:

    • Use a MachineSet to add labels to nodes managed by the MachineSet when a node is created:

      1. Run the following command to add a node selector to a MachineSet:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet abc612-msrtw-worker-us-east-1c  --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api
      2. Verify that the label is added to the MachineSet by using the oc edit command:

        For example:

        $ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-api
        Example MachineSet object
        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        
        ....
        
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
        ....
    • Add labels directly to a node:

      1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ip-10-0-142-25.ec2.internal type=user-node region=east
      2. Verify that the label is added to the node:

        $ oc get nodes -l type=user-node,region=east
        Example output
        NAME                          STATUS   ROLES    AGE   VERSION
        ip-10-0-142-25.ec2.internal   Ready    worker   17m   v1.18.3+002a51f
  2. Add the matching node selector a pod:

    • To add a node selector to existing and future pods, add a node selector to the controlling object for the pods:

      Example ReplicaSet object
      kind: ReplicaSet
      
      ....
      
      spec:
      
      ....
      
        template:
          metadata:
            creationTimestamp: null
            labels:
              ingresscontroller.operator.openshift.io/deployment-ingresscontroller: default
              pod-template-hash: 66d5cf9464
          spec:
            nodeSelector:
              beta.kubernetes.io/os: linux
              node-role.kubernetes.io/worker: ''
              type: user-node (1)
      1 Add the node selector.
    • To add a node selector to a specific pod, add the selector to the Pod object directly:

      Example Pod object
      apiVersion: v1
      kind: Pod
      
      ...
      
      spec:
        nodeSelector:
          <key>: <value>
      
      ...

      For example:

      Example Pod object with a node selector
      apiVersion: v1
      kind: Pod
      
      ....
      
      spec:
        nodeSelector:
          region: east
          type: user-node

Creating default cluster-wide node selectors

You can use default cluster-wide node selectors on pods together with labels on nodes to constrain all pods created in a cluster to specific nodes.

With cluster-wide node selectors, when you create a pod in that cluster, OpenShift Container Platform adds the default node selectors to the pod and schedules the pod on nodes with matching labels.

You configure cluster-wide node selectors by creating a Scheduler Operator custom resource (CR). You add labels to a node by editing a Node object, a MachineSet, or a MachineConfig. Adding the label to the MachineSet ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or MachineConfig do not persist if the node or machine goes down.

For example, the the Scheduler configures the cluster-wide region=east node selector:

Example Scheduler Operator Custom Resource
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
  name: cluster
...

spec:
  defaultNodeSelector: type=user-node,region=east (1)
...

A node in that cluster has the type=user-node,region=east labels:

Example Node object
apiVersion: v1
kind: Node
metadata:
  name: ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4
...
  labels:
    region: east
    type: user-node
...

If you create a pod in that cluster, the pod is created with the cluster-wide node selector and is scheduled on the labeled node:

Example Pod object
apiVersion: v1
kind: Pod
...

spec:
  nodeSelector:
    region: east
...
Example pod list with the pod on the labeled node
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
pod-s1   1/1     Running   0          20s   10.131.2.6   ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4   <none>           <none>

If the project where you create the pod has a project node selector, that selector takes preference over a cluster-wide node selector. Your pod is not created or scheduled if the node selector in the Pod spec does not use the project node selector.

A pod is not created or scheduled if the Pod object contains a node selector that is not the cluster-wide node selector or not a project node selector. For example, if you deploy the following pod into the example project, it will not be created:

Example Pod output with an invalid node selector
apiVersion: v1
kind: Pod
....

spec:
  nodeSelector:
    region: west

When you create a pod from that spec, you receive an error similar to the following message:

Example error message
Error from server (Forbidden): error when creating "pod.yaml": pods "pod-4" is forbidden: pod node label selector conflicts with its project node label selector

You can add additional key/value pairs to a pod. But you cannot add a different value for a default project key.

Procedure

To add a default cluster-wide node selector:

  1. Edit the Scheduler Operator custom resource to add the cluster node selectors:

    $ oc edit scheduler cluster
    apiVersion: config.openshift.io/v1
    kind: Scheduler
    metadata:
      name: cluster
    ...
    
    spec:
      defaultNodeSelector: type=user-node,region=east (1)
      mastersSchedulable: false
      policy:
        name: ""
    1 Add a node selector with the appropriate <key>:<value> pairs.

    After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster node selector does not take effect until the pods redeploy.

  2. Add labels to a node by using a MachineSet or editing the node directly:

    • Use a MachineSet to add labels to nodes managed by the MachineSet when a node is created:

      1. Run the following command to add a node selector to a MachineSet:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api (1)
        1 Add a <key>/<value> pair for each node selector.

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api
      2. Verify that the label is added to the MachineSet by using the oc edit command:

        For example:

        $ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
        Example output
        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
        ...
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
      3. Redeploy the nodes associated with that MachineSet by scaling down to 0 and scaling up the nodes:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
      4. When the node is ready and available, verify that the label is added to the node by using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node
        Example output
        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f
    • Add labels directly to a node:

      1. Edit the Node object for the node:

        $ oc label nodes <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east
      2. Verify that the label is added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>,<key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east
        Example output
        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f

Creating project-wide node selectors

You can use node selectors in a project together with labels on nodes to constrain all pods created in that project to the labeled nodes.

When you create a pod in this project, OpenShift Container Platform adds the node selectors to the pods in the project and schedules the pods on a node with matching labels in the project. If there is a cluster-wide default node selector, a project node selector takes preference.

You add labels to a project by editing the Namespace object to add the openshift.io/node-selector parameter, which contains the label definitions. You add labels to a node by editing the Node object, a MachineSet, or a MachineConfig. Adding the label to the MachineSet ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or MachineConfig do not persist if the node or machine goes down.

You can add additional key/value pairs to a pod. But you cannot add a different value for a default project key.

For example, the following project has the region=east node selector:

Example Namespace object
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/node-selector: "region=east"
...

The following node has the type=user-node,region=east labels:

Example Node object
apiVersion: v1
kind: Node
metadata:
  name: ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4
...
  labels:
    region: east
    type: user-node
...

If you create a pod in this example project, the pod is created with the project node selector and is scheduled on the labeled node:

Example Pod object
apiVersion: v1
kind: Pod
metadata:
...
spec:
  nodeSelector:
    region: east
    type: user-node
...
Example pod list with the pod on the labeled node
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE                                       NOMINATED NODE   READINESS GATES
pod-s1   1/1     Running   0          20s   10.131.2.6   ci-ln-qg1il3k-f76d1-hlmhl-worker-b-df2s4   <none>           <none>

A pod in the project is not created or scheduled if the pod contains different node selectors. For example, if you deploy the following pod into the example project, it will not be created:

Example Pod object with an invalid node selector
apiVersion: v1
kind: Pod
...

spec:
  nodeSelector:
    region: west

....

When you create the pod, you receive an error similar to the following message:

Example error message
Error from server (Forbidden): error when creating "pod.yaml": pods "pod-4" is forbidden: pod node label selector conflicts with its project node label selector
Procedure

To add a default project node selector:

  1. Create a project or edit an existing project to add the openshift.io/node-selector parameter:

    $ oc edit project <name>
    apiVersion: v1
    kind: Project
    metadata:
      annotations:
        openshift.io/node-selector: "type=user-node,region=east" (1)
        openshift.io/sa.scc.mcs: s0:c17,c14
        openshift.io/sa.scc.supplemental-groups: 1000300000/10000
        openshift.io/sa.scc.uid-range: 1000300000/10000
      creationTimestamp: 2019-06-10T14:39:45Z
      labels:
        openshift.io/run-level: "0"
      name: demo
      resourceVersion: "401885"
      selfLink: /api/v1/namespaces/openshift-kube-apiserver
      uid: 96ecc54b-8b8d-11e9-9f54-0a9ae641edd0
    spec:
      finalizers:
      - kubernetes
    status:
      phase: Active
    1 Add the openshift.io/node-selector with the appropriate <key>:<value> pairs.
  2. Add labels to a node by using a MachineSet or editing the node directly:

    • Use a MachineSet to add labels to nodes managed by the MachineSet when a node is created:

      1. Run the following command to add a node selector to a MachineSet:

        $ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]'  -n openshift-machine-api

        For example:

        $ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]'  -n openshift-machine-api
      2. Verify that the label is added to the MachineSet by using the oc edit command:

        For example:

        $ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
        Example output
        apiVersion: machine.openshift.io/v1beta1
        kind: MachineSet
        metadata:
        ...
        spec:
        ...
          template:
            metadata:
        ...
            spec:
              metadata:
                labels:
                  region: east
                  type: user-node
      3. Redeploy the nodes associated with that MachineSet:

        For example:

        $ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
        $ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api
      4. Verify that the label is added to the node, when the node is ready and available, using the oc get command:

        $ oc label MachineSet abc612-msrtw-worker-us-east-1c type=user-node region=east

        For example:

        $ oc get nodes -l type=user-node
        Example output
        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp   Ready    worker   61s   v1.18.3+002a51f
    • Add labels directly to a node:

      1. Edit the Node object to add labels:

        $ oc label <resource> <name> <key>=<value>

        For example, to label a node:

        $ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-c-tgq49 type=user-node region=east
      2. Verify that the label is added to the node using the oc get command:

        $ oc get nodes -l <key>=<value>

        For example:

        $ oc get nodes -l type=user-node,region=east
        Example output
        NAME                                       STATUS   ROLES    AGE   VERSION
        ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49   Ready    worker   17m   v1.18.3+002a51f