Controlling pod placement on nodes using node affinity rules - Controlling pod placement onto nodes (scheduling) | Nodes

Understanding node affinity
Configuring a required node affinity rule
Configuring a preferred node affinity rule
Sample node affinity rules
- Node affinity with matching labels
- Node affinity with no matching labels

Affinity is a property of pods that controls the nodes on which they prefer to be scheduled.

In Red Hat OpenShift Service on AWS node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The rules are defined using custom labels on the nodes and label selectors specified in pods.

edit

Understanding node affinity

Node affinity allows a pod to specify an affinity towards a group of nodes it can be placed on. The node does not have control over the placement.

For example, you could configure a pod to only run on a node with a specific CPU or in a specific availability zone.

There are two types of node affinity rules: required and preferred.

Required rules must be met before a pod can be scheduled on a node. Preferred rules specify that, if the rule is met, the scheduler tries to enforce the rules, but does not guarantee enforcement.

If labels on a node change at runtime that results in an node affinity rule on a pod no longer being met, the pod continues to run on the node.

You configure node affinity through the Pod spec file. You can specify a required rule, a preferred rule, or both. If you specify both, the node must first meet the required rule, then attempts to meet the preferred rule.

The following example is a Pod spec with a rule that requires the pod be placed on a node with a label whose key is e2e-az-NorthSouth and whose value is either e2e-az-North or e2e-az-South:

Example pod configuration file with a node affinity required rule

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  affinity:
    nodeAffinity: (1)
      requiredDuringSchedulingIgnoredDuringExecution: (2)
        nodeSelectorTerms:
        - matchExpressions:
          - key: e2e-az-NorthSouth (3)
            operator: In (4)
            values:
            - e2e-az-North (3)
            - e2e-az-South (3)
  containers:
  - name: with-node-affinity
    image: docker.io/ocpqe/hello-pod
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]
# ...

1	The stanza to configure node affinity.
2	Defines a required rule.
3	The key/value pair (label) that must be matched to apply the rule.
4	The operator represents the relationship between the label on the node and the set of values in the `matchExpression` parameters in the `Pod` spec. This value can be `In`, `NotIn`, `Exists`, or `DoesNotExist`, `Lt`, or `Gt`.

The following example is a node specification with a preferred rule that a node with a label whose key is e2e-az-EastWest and whose value is either e2e-az-East or e2e-az-West is preferred for the pod:

Example pod configuration file with a node affinity preferred rule

apiVersion: v1
kind: Pod
metadata:
  name: with-node-affinity
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  affinity:
    nodeAffinity: (1)
      preferredDuringSchedulingIgnoredDuringExecution: (2)
      - weight: 1 (3)
        preference:
          matchExpressions:
          - key: e2e-az-EastWest (4)
            operator: In (5)
            values:
            - e2e-az-East (4)
            - e2e-az-West (4)
  containers:
  - name: with-node-affinity
    image: docker.io/ocpqe/hello-pod
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]
# ...

1	The stanza to configure node affinity.
2	Defines a preferred rule.
3	Specifies a weight for a preferred rule. The node with highest weight is preferred.
4	The key/value pair (label) that must be matched to apply the rule.
5	The operator represents the relationship between the label on the node and the set of values in the `matchExpression` parameters in the `Pod` spec. This value can be `In`, `NotIn`, `Exists`, or `DoesNotExist`, `Lt`, or `Gt`.

There is no explicit node anti-affinity concept, but using the NotIn or DoesNotExist operator replicates that behavior.

If you are using node affinity and node selectors in the same pod configuration, note the following:

If you configure both nodeSelector and nodeAffinity, both conditions must be satisfied for the pod to be scheduled onto a candidate node.
If you specify multiple nodeSelectorTerms associated with nodeAffinity types, then the pod can be scheduled onto a node if one of the nodeSelectorTerms is satisfied.
If you specify multiple matchExpressions associated with nodeSelectorTerms, then the pod can be scheduled onto a node only if all matchExpressions are satisfied.

edit

Configuring a required node affinity rule

Required rules must be met before a pod can be scheduled on a node.

Procedure

The following steps demonstrate a simple configuration that creates a node and a pod that the scheduler is required to place on the node.

Create a pod with a specific label in the pod spec:

Create a YAML file with the following content:

You cannot add an affinity directly to a scheduled pod.

Example output

apiVersion: v1
kind: Pod
metadata:
  name: s1
spec:
  affinity: (1)
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution: (2)
        nodeSelectorTerms:
        - matchExpressions:
          - key: e2e-az-name (3)
            values:
            - e2e-az1
            - e2e-az2
            operator: In (4)
#...

1	Adds a pod affinity.
2	Configures the `requiredDuringSchedulingIgnoredDuringExecution` parameter.
3	Specifies the `key` and `values` that must be met. If you want the new pod to be scheduled on the node you edited, use the same `key` and `values` parameters as the label in the node.
4	Specifies an `operator`. The operator can be `In`, `NotIn`, `Exists`, or `DoesNotExist`. For example, use the operator `In` to require the label to be in the node.

Create the pod:
```
$ oc create -f <file-name>.yaml
```

edit

Configuring a preferred node affinity rule

Preferred rules specify that, if the rule is met, the scheduler tries to enforce the rules, but does not guarantee enforcement.

Procedure

The following steps demonstrate a simple configuration that creates a node and a pod that the scheduler tries to place on the node.

Create a pod with a specific label:

Create a YAML file with the following content:

You cannot add an affinity directly to a scheduled pod.

apiVersion: v1
kind: Pod
metadata:
  name: s1
spec:
  affinity: (1)
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution: (2)
      - weight: (3)
        preference:
          matchExpressions:
          - key: e2e-az-name (4)
            values:
            - e2e-az3
            operator: In (5)
#...

1	Adds a pod affinity.
2	Configures the `preferredDuringSchedulingIgnoredDuringExecution` parameter.
3	Specifies a weight for the node, as a number 1-100. The node with highest weight is preferred.
4	Specifies the `key` and `values` that must be met. If you want the new pod to be scheduled on the node you edited, use the same `key` and `values` parameters as the label in the node.
5	Specifies an `operator`. The operator can be `In`, `NotIn`, `Exists`, or `DoesNotExist`. For example, use the operator `In` to require the label to be in the node.

Create the pod.
```
$ oc create -f <file-name>.yaml
```

Sample node affinity rules

The following examples demonstrate node affinity.

edit

Node affinity with matching labels

The following example demonstrates node affinity for a node and pod with matching labels:

The Node1 node has the label zone:us:

$ oc label node node1 zone=us

You can alternatively apply the following YAML to add the label:

kind: Node
apiVersion: v1
metadata:
  name: <node_name>
  labels:
    zone: us
#...

The pod-s1 pod has the zone and us key/value pair under a required node affinity rule:

$ cat pod-s1.yaml

Example output

apiVersion: v1
kind: Pod
metadata:
  name: pod-s1
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - image: "docker.io/ocpqe/hello-pod"
      name: hello-pod
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: "zone"
              operator: In
              values:
              - us
#...

The pod-s1 pod can be scheduled on Node1:

$ oc get pod -o wide

Example output

NAME     READY     STATUS       RESTARTS   AGE      IP      NODE
pod-s1   1/1       Running      0          4m       IP1     node1

Node affinity with no matching labels

The following example demonstrates node affinity for a node and pod without matching labels:

The Node1 node has the label zone:emea:

$ oc label node node1 zone=emea

You can alternatively apply the following YAML to add the label:

kind: Node
apiVersion: v1
metadata:
  name: <node_name>
  labels:
    zone: emea
#...

The pod-s1 pod has the zone and us key/value pair under a required node affinity rule:

$ cat pod-s1.yaml

Example output

apiVersion: v1
kind: Pod
metadata:
  name: pod-s1
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - image: "docker.io/ocpqe/hello-pod"
      name: hello-pod
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: [ALL]
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: "zone"
              operator: In
              values:
              - us
#...

The pod-s1 pod cannot be scheduled on Node1:

$ oc describe pod pod-s1

Example output

...

Events:
 FirstSeen LastSeen Count From              SubObjectPath  Type                Reason
 --------- -------- ----- ----              -------------  --------            ------
 1m        33s      8     default-scheduler Warning        FailedScheduling    No nodes are available that match all of the following predicates:: MatchNodeSelector (1).