kubeletConfig: podsPerCore: 10
This topic provides recommended host practices for OpenShift Container Platform.
The OpenShift Container Platform node configuration file contains important options. For
example, two parameters control the maximum number of pods that can be scheduled
to a node:
When both options are in use, the lower of the two values limits the number of pods on a node. Exceeding these values can result in:
Increased CPU utilization.
Slow pod scheduling.
Potential out-of-memory scenarios, depending on the amount of memory in the node.
Exhausting the pool of IP addresses.
Resource overcommitting, leading to poor user application performance.
In Kubernetes, a pod that is holding a single container actually uses two containers. The second container is used to set up networking prior to the actual container starting. Therefore, a system running 10 pods will actually have 20 containers running.
podsPerCore sets the number of pods the node can run based on the number of
processor cores on the node. For example, if
podsPerCore is set to
10 on a
node with 4 processor cores, the maximum number of pods allowed on the node will
kubeletConfig: podsPerCore: 10
0 disables this limit. The default is
podsPerCore cannot exceed
maxPods sets the number of pods the node can run to a fixed value, regardless
of the properties of the node.
kubeletConfig: maxPods: 250
The kubelet configuration is currently serialized as an ignition configuration, so it can be directly edited. However, there is also a new kubelet-config-controller added to the Machine Config Controller (MCC). This allows you to create a KubeletConfig custom resource (CR) to edit the kubelet parameters.
$ oc get machineconfig
This provides a list of the available machine configuration objects you can
select. By default, the two kubelet-related configs are
To check the current value of max Pods per node, run:
# oc describe node <node-ip> | grep Allocatable -A6
value: pods: <value>.
# oc describe node ip-172-31-128-158.us-east-2.compute.internal | grep Allocatable -A6 Allocatable: attachable-volumes-aws-ebs: 25 cpu: 3500m hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 15341844Ki pods: 250
To set the max Pods per node on the worker nodes, create a YAML file that
contains the kubelet configuration. For example,
apiVersion: machineconfiguration.openshift.io/v1 kind: KubeletConfig metadata: name: set-max-pods spec: machineConfigSelector: 01-worker-kubelet kubeletConfig: maxPods: 250
Apply the file by running:
$ oc apply -f max-worker-pods.yaml
Verify the change by running:
$ oc get kubeletconfigs set-max-pods -o yaml
By default, only one machine is allowed to be unavailable when applying the kubelet-related configuration to the available worker nodes. For a large cluster, it can take a long time for the configuration change to be reflected. At any time, you can adjust the number of machines that are updating to speed up the process.
$ oc edit machineconfigpool worker
maxUnavailable to the desired value.
spec: maxUnavailable: <node_count>
When setting the value, consider the number of worker nodes that can be unavailable without affecting the applications running on the cluster.
For large and dense clusters, etcd can suffer from poor performance
if the keyspace grows excessively large and exceeds the space quota.
Periodic maintenance of etcd including defragmentation needs to be done
to free up space in the data store. It is highly recommended that you monitor
Prometheus for etcd metrics and defragment it when needed before etcd raises
a cluster-wide alarm that puts the cluster into a maintenance mode, which
only accepts key reads and deletes. Some of the key metrics to monitor are
etcd_server_quota_backend_bytes which is the current quota limit,
etcd_mvcc_db_total_size_in_use_in_bytes which indicates the actual
database usage after a history compaction, and
etcd_debugging_mvcc_db_total_size_in_bytes which shows the database size
including free space waiting for defragmentation.