Soon, all Red Hat OpenShift documentation will only be available on docs.redhat.com, the official location of all Red Hat product documentation. Get familiar with the updated experience today!
  1. Documentation
    2. history
×

Using tolerations to control cluster logging pod placement

You can use taints and tolerations to ensure that cluster logging pods run on specific nodes and that no other workload can run on those nodes.

Taints and tolerations are simple key:value pair. A taint on a node instructs the node to repel all Pods that do not tolerate the taint.

The key is any string, up to 253 characters and the value is any string up to 63 characters. The string must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

Sample cluster logging CR with tolerations
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: openshift-logging
spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    elasticsearch:
      nodeCount: 1
      tolerations: (1)
      - key: "logging"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 6000
      resources:
        limits:
          memory: 8Gi
        requests:
          cpu: 100m
          memory: 1Gi
      storage: {}
      redundancyPolicy: "ZeroRedundancy"
  visualization:
    type: "kibana"
    kibana:
      tolerations: (2)
      - key: "logging"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 6000
      resources:
        limits:
          memory: 2Gi
        requests:
          cpu: 100m
          memory: 1Gi
      replicas: 1
  curation:
    type: "curator"
    curator:
      tolerations: (3)
      - key: "logging"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 6000
      resources:
        limits:
          memory: 200Mi
        requests:
          cpu: 100m
          memory: 100Mi
      schedule: "*/5 * * * *"
  collection:
    logs:
      type: "fluentd"
      fluentd:
        tolerations: (4)
        - key: "logging"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 6000
        resources:
          limits:
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 1Gi
1 This toleration is added to the Elasticsearch pods.
2 This toleration is added to the Kibana pod.
3 This toleration is added to the Curator pod.
4 This toleration is added to the logging collector pods.

Using tolerations to control the Elasticsearch Pod placement

You can control which nodes the Elasticsearch Pods runs on and prevent other workloads from using those nodes by using tolerations on the Pods.

You apply tolerations to Elasticsearch Pods through the Cluster Logging Custom Resource (CR) and apply taints to a node through the node specification. A taint on a node is a key:value pair that instructs the node to repel all Pods that do not tolerate the taint. Using a specific key:value pair that is not on other Pods ensures only Elasticseach Pods can run on that node.

By default, the Elasticsearch Pods have the following toleration:

tolerations:
- effect: "NoExecute"
  key: "node.kubernetes.io/disk-pressure"
  operator: "Exists"
Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want to schedule the cluster logging Pods:

    $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    $ oc adm taint nodes node1 elasticsearch=node:NoExecute

    This example places a taint on node1 that has key elasticsearch, value node, and taint effect NoExecute. Nodes with the NoExecute effect schedule only Pods that match the taint and remove existing Pods that do not match.

  2. Edit the logstore section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Elasticsearch Pods:

      logStore:
    type: "elasticsearch"
    elasticsearch:
      nodeCount: 1
      tolerations:
      - key: "elasticsearch"  (1)
        operator: "Exists"  (2)
        effect: "NoExecute"  (3)
        tolerationSeconds: 6000  (4)
    1 Specify the key that you added to the node.
    2 Specify the Exists operator to require a taint with the key elasticsearch to be present on the Node.
    3 Specify the NoExecute effect.
    4 Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A Pod with this toleration could be scheduled onto node1.

Using tolerations to control the Kibana Pod placement

You can control which nodes the Kibana Pods run on and prevent other workloads from using those nodes by using tolerations on the Pods.

You apply tolerations to the Kibana Pods through the Cluster Logging Custom Resource (CR) and apply taints to a node through the node specification. A taint on a node is a key:value pair that instructs the node to repel all Pods that do not tolerate the taint. Using a specific key:value pair that is not on other Pods ensures only the Kibana Pod can run on that node.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want to schedule the Kibana Pod:

    $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    $ oc adm taint nodes node1 kibana=node:NoExecute

    This example places a taint on node1 that has key kibana, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

  2. Edit the visualization section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Kibana Pod:

      visualization:
    type: "kibana"
    kibana:
      tolerations:
      - key: "kibana"  (1)
        operator: "Exists"  (2)
        effect: "NoExecute"  (3)
        tolerationSeconds: 6000 (4)
    1 Specify the key that you added to the node.
    2 Specify the Exists operator to require the key/value/effect parameters to match.
    3 Specify the NoExecute effect.
    4 Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A Pod with this toleration would be able to schedule onto node1.

Using tolerations to control the Curator Pod placement

You can control which node the Curator Pod runs on and prevent other workloads from using those nodes by using tolerations on the Pod.

You apply tolerations to the Curator Pod through the Cluster Logging Custom Resource (CR) and apply taints to a node through the node specification. A taint on a node is a key:value pair that instructs the node to repel all Pods that do not tolerate the taint. Using a specific key:value pair that is not on other Pods ensures only the Curator Pod can run on that node.

Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want to schedule the Curator Pod:

    $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    $ oc adm taint nodes node1 curator=node:NoExecute

    This example places a taint on node1 that has key curator, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and remove existing Pods that do not match.

  2. Edit the curation section of the Cluster Logging Custom Resource (CR) to configure a toleration for the Curator Pod:

      curation:
    type: "curator"
    curator:
      tolerations:
      - key: "curator"  (1)
        operator: "Exists"  (2)
        effect: "NoExecute"  (3)
        tolerationSeconds: 6000  (4)
    1 Specify the key that you added to the node.
    2 Specify the Exists operator to require the key/value/effect parameters to match.
    3 Specify the NoExecute effect.
    4 Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

This toleration matches the taint that is created by the oc adm taint command. A Pod with this toleration would be able to schedule onto node1.

Using tolerations to control the log collector Pod placement

You can ensure which nodes the logging collector Pods run on and prevent other workloads from using those nodes by using tolerations on the Pods.

You apply tolerations to logging collector Pods through the Cluster Logging Custom Resource (CR) and apply taints to a node through the node specification. You can use taints and tolerations to ensure the Pod does not get evicted for things like memory and CPU issues.

By default, the logging collector Pods have the following toleration:

tolerations:
- key: "node-role.kubernetes.io/master"
  operator: "Exists"
  effect: "NoExecute"
Prerequisites

  • Cluster logging and Elasticsearch must be installed.

Procedure

  1. Use the following command to add a taint to a node where you want logging collector Pods to schedule logging collector Pods:

    $ oc adm taint nodes <node-name> <key>=<value>:<effect>

    For example:

    $ oc adm taint nodes node1 collector=node:NoExecute

    This example places a taint on node1 that has key collector, value node, and taint effect NoExecute. You must use the NoExecute taint effect. NoExecute schedules only Pods that match the taint and removes existing Pods that do not match.

  2. Edit the collection section of the Cluster Logging Custom Resource (CR) to configure a toleration for the logging collector Pods:

      collection:
    logs:
      type: "fluentd"
      rsyslog:
        tolerations:
        - key: "collector"  (1)
          operator: "Exists"  (2)
          effect: "NoExecute"  (3)
          tolerationSeconds: 6000  (4)
    1 Specify the key that you added to the node.
    2 Specify the Exists operator to require the key/value/effect parameters to match.
    3 Specify the NoExecute effect.
    4 Optionally, specify the tolerationSeconds parameter to set how long a Pod can remain bound to a node before being evicted.

This toleration matches the taint created by the oc adm taint command. A Pod with this toleration would be able to schedule onto node1.

Additional resources

For more information about taints and tolerations, see Controlling pod placement using node taints.