apiserver.config.openshift.io
After installing OpenShift Container Platform, you can further expand and customize your cluster to your requirements.
You complete most of the cluster configuration and customization after you deploy your OpenShift Container Platform cluster. A number of configuration resources are available.
|
If you install your cluster on IBM Z, not all features and functions are available. |
You modify the configuration resources to configure the major features of the cluster, such as the image registry, networking configuration, image build behavior, and the identity provider.
For current documentation of the settings that you control by using these resources, use
the oc explain command, for example oc explain builds --api-version=config.openshift.io/v1
All cluster configuration resources are globally scoped (not namespaced) and named cluster.
| Resource name | Description |
|---|---|
|
Provides API server configuration such as certificates and certificate authorities. |
|
Controls the identity provider and authentication configuration for the cluster. |
|
Controls default and enforced configuration for all builds on the cluster. |
|
Configures the behavior of the web console interface, including the logout behavior. |
|
Enables FeatureGates so that you can use Tech Preview features. |
|
Configures how specific image registries should be treated (allowed, disallowed, insecure, CA details). |
|
Configuration details related to routing such as the default domain for routes. |
|
Configures identity providers and other behavior related to internal OAuth server flows. |
|
Configures how projects are created including the project template. |
|
Defines proxies to be used by components needing external network access. Note: not all components currently consume this value. |
|
Configures scheduler behavior such as policies and default node selectors. |
These configuration resources are cluster-scoped instances, named cluster, which control the behavior of a specific component as
owned by a particular Operator.
| Resource name | Description |
|---|---|
|
Controls console appearance such as branding customizations |
|
Configures internal image registry settings such as public routing, log levels, proxy settings, resource constraints, replica counts, and storage type. |
|
Configures the Samples Operator to control which example image streams and templates are installed on the cluster. |
These configuration resources represent a single instance of a particular component. In some cases, you can request multiple instances by creating multiple instances of the resource. In other cases, the Operator can use only a specific resource instance name in a specific namespace. Reference the component-specific documentation for details on how and when you can create additional resource instances.
| Resource name | Instance name | Namespace | Description |
|---|---|---|---|
|
|
|
Controls the Alertmanager deployment parameters. |
|
|
|
Configures Ingress Operator behavior such as domain, number of replicas, certificates, and controller placement. |
You use these resources to retrieve information about the cluster. Some configurations might require you to edit these resources directly.
| Resource name | Instance name | Description |
|---|---|---|
|
|
In OpenShift Container Platform 4.9, you must not customize the |
|
|
You cannot modify the DNS settings for your cluster. You can view the DNS Operator status. |
|
|
Configuration details allowing the cluster to interact with its cloud provider. |
|
|
You cannot modify your cluster networking after installation. To customize your network, follow the process to customize networking during installation. |
You can update the global pull secret for your cluster by either replacing the current pull secret or appending a new pull secret.
The procedure is required when users use a separate registry to store images than the registry used during installation.
|
Cluster resources must adjust to the new pull secret, which can temporarily limit the usability of the cluster. |
You have access to the cluster as a user with the cluster-admin role.
Optional: To append a new pull secret to the existing pull secret, complete the following steps:
Enter the following command to download the pull secret:
$ oc get secret/pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' ><pull_secret_location> (1)| 1 | Provide the path to the pull secret file. |
Enter the following command to add the new pull secret:
$ oc registry login --registry="<registry>" \ (1)
--auth-basic="<username>:<password>" \ (2)
--to=<pull_secret_location> (3)
| 1 | Provide the new registry. You can include multiple repositories within the same registry, for example: --registry="<registry/my-namespace/my-repository>". |
| 2 | Provide the credentials of the new registry. |
| 3 | Provide the path to the pull secret file. |
Alternatively, you can perform a manual update to the pull secret file.
Enter the following command to update the global pull secret for your cluster:
$ oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=<pull_secret_location> (1)| 1 | Provide the path to the new pull secret file. |
This update is rolled out to all nodes, which can take some time depending on the size of your cluster.
|
As of OpenShift Container Platform 4.7.4, changes to the global pull secret no longer trigger a node drain or reboot. |
If you incorrectly sized the worker nodes during deployment, adjust them by creating one or more new machine sets, scale them up, then scale the original machine set down before removing them.
MachineSet objects describe OpenShift Container Platform nodes with respect to the cloud or machine provider.
The MachineConfigPool object allows MachineConfigController components to define and provide the status of machines in the context of upgrades.
The MachineConfigPool object allows users to configure how upgrades are rolled out to the OpenShift Container Platform nodes in the machine config pool.
The NodeSelector object can be replaced with a reference to the MachineSet object.
To add or remove an instance of a machine in a machine set, you can manually scale the machine set.
This guidance is relevant to fully automated, installer-provisioned infrastructure installations. Customized, user-provisioned infrastructure installations do not have machine sets.
Install an OpenShift Container Platform cluster and the oc command line.
Log in to oc as a user with cluster-admin permission.
View the machine sets that are in the cluster:
$ oc get machinesets -n openshift-machine-apiThe machine sets are listed in the form of <clusterid>-worker-<aws-region-az>.
View the machines that are in the cluster:
$ oc get machine -n openshift-machine-apiSet the annotation on the machine that you want to delete:
$ oc annotate machine/<machine_name> -n openshift-machine-api machine.openshift.io/cluster-api-delete-machine="true"Cordon and drain the node that you want to delete:
$ oc adm cordon <node_name>
$ oc adm drain <node_name>Scale the machine set:
$ oc scale --replicas=2 machineset <machineset> -n openshift-machine-apiOr:
$ oc edit machineset <machineset> -n openshift-machine-api|
You can alternatively apply the following YAML to scale the machine set: |
You can scale the machine set up or down. It takes several minutes for the new machines to be available.
Verify the deletion of the intended machine:
$ oc get machinesRandom, Newest, and Oldest are the three supported deletion options. The default is Random, meaning that random machines are chosen and deleted when scaling machine sets down. The deletion policy can be set according to the use case by modifying the particular machine set:
spec:
deletePolicy: <delete_policy>
replicas: <desired_replica_count>Specific machines can also be prioritized for deletion by adding the annotation machine.openshift.io/cluster-api-delete-machine to the machine of interest, regardless of the deletion policy.
|
By default, the OpenShift Container Platform router pods are deployed on workers. Because the router is required to access some cluster resources, including the web console, do not scale the worker machine set to |
|
Custom machine sets can be used for use cases requiring that services run on specific nodes and that those services are ignored by the controller when the worker machine sets are scaling down. This prevents service disruption. |
You can use default cluster-wide node selectors on pods together with labels on nodes to constrain all pods created in a cluster to specific nodes.
With cluster-wide node selectors, when you create a pod in that cluster, OpenShift Container Platform adds the default node selectors to the pod and schedules the pod on nodes with matching labels.
You configure cluster-wide node selectors by editing the Scheduler Operator custom resource (CR). You add labels to a node, a machine set, or a machine config. Adding the label to the machine set ensures that if the node or machine goes down, new nodes have the label. Labels added to a node or machine config do not persist if the node or machine goes down.
|
You can add additional key/value pairs to a pod. But you cannot add a different value for a default key. |
To add a default cluster-wide node selector:
Edit the Scheduler Operator CR to add the default cluster-wide node selectors:
$ oc edit scheduler clusterapiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
name: cluster
...
spec:
defaultNodeSelector: type=user-node,region=east (1)
mastersSchedulable: false
policy:
name: ""| 1 | Add a node selector with the appropriate <key>:<value> pairs. |
After making this change, wait for the pods in the openshift-kube-apiserver project to redeploy. This can take several minutes. The default cluster-wide node selector does not take effect until the pods redeploy.
Add labels to a node by using a machine set or editing the node directly:
Use a machine set to add labels to nodes managed by the machine set when a node is created:
Run the following command to add labels to a MachineSet object:
$ oc patch MachineSet <name> --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"<key>"="<value>","<key>"="<value>"}}]' -n openshift-machine-api (1)| 1 | Add a <key>/<value> pair for each label. |
For example:
$ oc patch MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c --type='json' -p='[{"op":"add","path":"/spec/template/spec/metadata/labels", "value":{"type":"user-node","region":"east"}}]' -n openshift-machine-api|
You can alternatively apply the following YAML to add labels to a machine set: |
Verify that the labels are added to the MachineSet object by using the oc edit command:
For example:
$ oc edit MachineSet abc612-msrtw-worker-us-east-1c -n openshift-machine-apiMachineSet objectapiVersion: machine.openshift.io/v1beta1
kind: MachineSet
...
spec:
...
template:
metadata:
...
spec:
metadata:
labels:
region: east
type: user-node
...Redeploy the nodes associated with that machine set by scaling down to 0 and scaling up the nodes:
For example:
$ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api$ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-apiVerify that the labels are added to the MachineSet object by using the oc edit command:
For example:
$ oc edit MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-apiRedeploy the nodes associated with that machine set by scaling down to 0 and scaling up the nodes:
For example:
$ oc scale --replicas=0 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-api$ oc scale --replicas=1 MachineSet ci-ln-l8nry52-f76d1-hl7m7-worker-c -n openshift-machine-apiWhen the nodes are ready and available, verify that the label is added to the nodes by using the oc get command:
$ oc get nodes -l <key>=<value>For example:
$ oc get nodes -l type=user-nodeNAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-c-vmqzp Ready worker 61s v1.22.1Add labels directly to a node:
Edit the Node object for the node:
$ oc label nodes <name> <key>=<value>For example, to label a node:
$ oc label nodes ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 type=user-node region=east|
You can alternatively apply the following YAML to add labels to a node: |
Verify that the labels are added to the node using the oc get command:
$ oc get nodes -l <key>=<value>,<key>=<value>For example:
$ oc get nodes -l type=user-node,region=eastNAME STATUS ROLES AGE VERSION
ci-ln-l8nry52-f76d1-hl7m7-worker-b-tgq49 Ready worker 17m v1.22.1You can create a machine set to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment.
In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. Both OpenShift Logging and Red Hat OpenShift Service Mesh deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. A configuration like this requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.
For information on infrastructure nodes and which components can run on infrastructure nodes, see Creating infrastructure machine sets.
For sample machine sets that you can use with these procedures, see Creating machine sets for different clouds.
In addition to the ones created by the installation program, you can create your own machine sets to dynamically manage the machine compute resources for specific workloads of your choice.
Deploy an OpenShift Container Platform cluster.
Install the OpenShift CLI (oc).
Log in to oc as a user with cluster-admin permission.
Create a new YAML file that contains the machine set custom resource (CR) sample and is named <file_name>.yaml.
Ensure that you set the <clusterID> and <role> parameter values.
If you are not sure which value to set for a specific field, you can check an existing machine set from your cluster:
$ oc get machinesets -n openshift-machine-apiNAME DESIRED CURRENT READY AVAILABLE AGE
agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1d 0 0 55m
agl030519-vplxk-worker-us-east-1e 0 0 55m
agl030519-vplxk-worker-us-east-1f 0 0 55mCheck values of a specific machine set:
$ oc get machineset <machineset_name> -n \
openshift-machine-api -o yaml...
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: agl030519-vplxk (1)
machine.openshift.io/cluster-api-machine-role: worker (2)
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a| 1 | The cluster ID. |
| 2 | A default node label. |
Create the new MachineSet CR:
$ oc create -f <file_name>.yamlView the list of machine sets:
$ oc get machineset -n openshift-machine-apiNAME DESIRED CURRENT READY AVAILABLE AGE
agl030519-vplxk-infra-us-east-1a 1 1 1 1 11m
agl030519-vplxk-worker-us-east-1a 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1b 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1c 1 1 1 1 55m
agl030519-vplxk-worker-us-east-1d 0 0 55m
agl030519-vplxk-worker-us-east-1e 0 0 55m
agl030519-vplxk-worker-us-east-1f 0 0 55mWhen the new machine set is available, the DESIRED and CURRENT values match. If the machine set is not available, wait a few minutes and run the command again.
|
See Creating infrastructure machine sets for installer-provisioned infrastructure environments or for any cluster where the control plane nodes are managed by the machine API. |
Requirements of the cluster dictate that infrastructure, also called infra nodes, be provisioned. The installer only provides provisions for control plane and worker nodes. Worker nodes can be designated as infrastructure nodes or application, also called app, nodes through labeling.
Add a label to the worker node that you want to act as application node:
$ oc label node <node-name> node-role.kubernetes.io/app=""Add a label to the worker nodes that you want to act as infrastructure nodes:
$ oc label node <node-name> node-role.kubernetes.io/infra=""Check to see if applicable nodes now have the infra role and app roles:
$ oc get nodesCreate a default cluster-wide node selector. The default node selector is applied to pods created in all namespaces. This creates an intersection with any existing node selectors on a pod, which additionally constrains the pod’s selector.
|
If the default node selector key conflicts with the key of a pod’s label, then the default node selector is not applied. However, do not set a default node selector that might cause a pod to become unschedulable. For example, setting the default node selector to a specific node role, such as You can alternatively use a project node selector to avoid cluster-wide node selector key conflicts. |
Edit the Scheduler object:
$ oc edit scheduler clusterAdd the defaultNodeSelector field with the appropriate node selector:
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
name: cluster
...
spec:
defaultNodeSelector: topology.kubernetes.io/region=us-east-1 (1)
...| 1 | This example node selector deploys pods on nodes in the us-east-1 region by default. |
Save the file to apply the changes.
You can now move infrastructure resources to the newly labeled infra nodes.
For information on how to configure project node selectors to avoid cluster-wide node selector key conflicts, see Project node selectors.
If you need infrastructure machines to have dedicated configurations, you must create an infra pool.
Add a label to the node you want to assign as the infra node with a specific label:
$ oc label node <node_name> <label>$ oc label node ci-ln-n8mqwr2-f76d1-xscn2-worker-c-6fmtx node-role.kubernetes.io/infra=Create a machine config pool that contains both the worker role and your custom role as machine config selector:
$ cat infra.mcp.yamlapiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: infra
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]} (1)
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: "" (2)| 1 | Add the worker role and your custom role. |
| 2 | Add the label you added to the node as a nodeSelector. |
|
Custom machine config pools inherit machine configs from the worker pool. Custom pools use any machine config targeted for the worker pool, but add the ability to also deploy changes that are targeted at only the custom pool. Because a custom pool inherits resources from the worker pool, any change to the worker pool also affects the custom pool. |
After you have the YAML file, you can create the machine config pool:
$ oc create -f infra.mcp.yamlCheck the machine configs to ensure that the infrastructure configuration rendered successfully:
$ oc get machineconfigNAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED
00-master 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
00-worker 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-master-container-runtime 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-master-kubelet 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-worker-container-runtime 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
01-worker-kubelet 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
99-master-1ae2a1e0-a115-11e9-8f14-005056899d54-registries 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
99-master-ssh 3.2.0 31d
99-worker-1ae64748-a115-11e9-8f14-005056899d54-registries 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 31d
99-worker-ssh 3.2.0 31d
rendered-infra-4e48906dca84ee702959c71a53ee80e7 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 23m
rendered-master-072d4b2da7f88162636902b074e9e28e 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-master-3e88ec72aed3886dec061df60d16d1af 02c07496ba0417b3e12b78fb32baf6293d314f79 3.2.0 31d
rendered-master-419bee7de96134963a15fdf9dd473b25 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 17d
rendered-master-53f5c91c7661708adce18739cc0f40fb 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 13d
rendered-master-a6a357ec18e5bce7f5ac426fc7c5ffcd 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 7d3h
rendered-master-dc7f874ec77fc4b969674204332da037 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-worker-1a75960c52ad18ff5dfa6674eb7e533d 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-worker-2640531be11ba43c61d72e82dc634ce6 5b6fb8349a29735e48446d435962dec4547d3090 3.2.0 31d
rendered-worker-4e48906dca84ee702959c71a53ee80e7 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 7d3h
rendered-worker-4f110718fe88e5f349987854a1147755 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 17d
rendered-worker-afc758e194d6188677eb837842d3b379 02c07496ba0417b3e12b78fb32baf6293d314f79 3.2.0 31d
rendered-worker-daa08cc1e8f5fcdeba24de60cd955cc3 365c1cfd14de5b0e3b85e0fc815b0060f36ab955 3.2.0 13dYou should see a new machine config, with the rendered-infra-* prefix.
Optional: To deploy changes to a custom pool, create a machine config that uses the custom pool name as the label, such as infra. Note that this is not required and only shown for instructional purposes. In this manner, you can apply any custom configurations specific to only your infra nodes.
|
After you create the new machine config pool, the MCO generates a new rendered config for that pool, and associated nodes of that pool reboot to apply the new configuration. |
Create a machine config:
$ cat infra.mc.yamlapiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 51-infra
labels:
machineconfiguration.openshift.io/role: infra (1)
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- path: /etc/infratest
mode: 0644
contents:
source: data:,infra| 1 | Add the label you added to the node as a nodeSelector. |
Apply the machine config to the infra-labeled nodes:
$ oc create -f infra.mc.yamlConfirm that your new machine config pool is available:
$ oc get mcpNAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
infra rendered-infra-60e35c2e99f42d976e084fa94da4d0fc True False False 1 1 1 0 4m20s
master rendered-master-9360fdb895d4c131c7c4bebbae099c90 True False False 3 3 3 0 91m
worker rendered-worker-60e35c2e99f42d976e084fa94da4d0fc True False False 2 2 2 0 91mIn this example, a worker node was changed to an infra node.
See Node configuration management with machine config pools for more information on grouping infra machines in a custom pool.
After creating an infrastructure machine set, the worker and infra roles are applied to new infra nodes. Nodes with the infra role are not counted toward the total number of subscriptions that are required to run the environment, even when the worker role is also applied.
However, when an infra node is assigned the worker role, there is a chance that user workloads can get assigned inadvertently to the infra node. To avoid this, you can apply a taint to the infra node and tolerations for the pods that you want to control.
If you have an infra node that has the infra and worker roles assigned, you must configure the node so that user workloads are not assigned to it.
|
It is recommended that you preserve the dual |
Configure additional MachineSet objects in your OpenShift Container Platform cluster.
Add a taint to the infra node to prevent scheduling user workloads on it:
Determine if the node has the taint:
$ oc describe nodes <node_name>oc describe node ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
Name: ci-ln-iyhx092-f76d1-nvdfm-worker-b-wln2l
Roles: worker
...
Taints: node-role.kubernetes.io/infra:NoSchedule
...This example shows that the node has a taint. You can proceed with adding a toleration to your pod in the next step.
If you have not configured a taint to prevent scheduling user workloads on it:
$ oc adm taint nodes <node_name> <key>:<effect>For example:
$ oc adm taint nodes node1 node-role.kubernetes.io/infra:NoSchedule|
You can alternatively apply the following YAML to add the taint: |
This example places a taint on node1 that has key node-role.kubernetes.io/infra and taint effect NoSchedule. Nodes with the NoSchedule effect schedule only pods that tolerate the taint, but allow existing pods to remain scheduled on the node.
|
If a descheduler is used, pods violating node taints could be evicted from the cluster. |
Add tolerations for the pod configurations you want to schedule on the infra node, like router, registry, and monitoring workloads. Add the following code to the Pod object specification:
tolerations:
- effect: NoSchedule (1)
key: node-role.kubernetes.io/infra (2)
operator: Exists (3)| 1 | Specify the effect that you added to the node. |
| 2 | Specify the key that you added to the node. |
| 3 | Specify the Exists Operator to require a taint with the key node-role.kubernetes.io/infra to be present on the node. |
This toleration matches the taint created by the oc adm taint command. A pod with this toleration can be scheduled onto the infra node.
|
Moving pods for an Operator installed via OLM to an infra node is not always possible. The capability to move Operator pods depends on the configuration of each Operator. |
Schedule the pod to the infra node using a scheduler. See the documentation for Controlling pod placement onto nodes for details.
See Controlling pod placement using the scheduler for general information on scheduling a pod to a node.
Some of the infrastructure resources are deployed in your cluster by default. You can move them to the infrastructure machine sets that you created.
You can deploy the router pod to a different machine set. By default, the pod is deployed to a worker node.
Configure additional machine sets in your OpenShift Container Platform cluster.
View the IngressController custom resource for the router Operator:
$ oc get ingresscontroller default -n openshift-ingress-operator -o yamlThe command output resembles the following text:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
creationTimestamp: 2019-04-18T12:35:39Z
finalizers:
- ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
generation: 1
name: default
namespace: openshift-ingress-operator
resourceVersion: "11341"
selfLink: /apis/operator.openshift.io/v1/namespaces/openshift-ingress-operator/ingresscontrollers/default
uid: 79509e05-61d6-11e9-bc55-02ce4781844a
spec: {}
status:
availableReplicas: 2
conditions:
- lastTransitionTime: 2019-04-18T12:36:15Z
status: "True"
type: Available
domain: apps.<cluster>.example.com
endpointPublishingStrategy:
type: LoadBalancerService
selector: ingresscontroller.operator.openshift.io/deployment-ingresscontroller=defaultEdit the ingresscontroller resource and change the nodeSelector to use the infra label:
$ oc edit ingresscontroller default -n openshift-ingress-operatorAdd the nodeSelector stanza that references the infra label to the spec section, as shown:
spec:
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/infra: ""Confirm that the router pod is running on the infra node.
View the list of router pods and note the node name of the running pod:
$ oc get pod -n openshift-ingress -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-86798b4b5d-bdlvd 1/1 Running 0 28s 10.130.2.4 ip-10-0-217-226.ec2.internal <none> <none>
router-default-955d875f4-255g8 0/1 Terminating 0 19h 10.129.2.4 ip-10-0-148-172.ec2.internal <none> <none>In this example, the running pod is on the ip-10-0-217-226.ec2.internal node.
View the node status of the running pod:
$ oc get node <node_name> (1)| 1 | Specify the <node_name> that you obtained from the pod list. |
NAME STATUS ROLES AGE VERSION
ip-10-0-217-226.ec2.internal Ready infra,worker 17h v1.22.1Because the role list includes infra, the pod is running on the correct node.
You configure the registry Operator to deploy its pods to different nodes.
Configure additional machine sets in your OpenShift Container Platform cluster.
View the config/instance object:
$ oc get configs.imageregistry.operator.openshift.io/cluster -o yamlapiVersion: imageregistry.operator.openshift.io/v1
kind: Config
metadata:
creationTimestamp: 2019-02-05T13:52:05Z
finalizers:
- imageregistry.operator.openshift.io/finalizer
generation: 1
name: cluster
resourceVersion: "56174"
selfLink: /apis/imageregistry.operator.openshift.io/v1/configs/cluster
uid: 36fd3724-294d-11e9-a524-12ffeee2931b
spec:
httpSecret: d9a012ccd117b1e6616ceccb2c3bb66a5fed1b5e481623
logging: 2
managementState: Managed
proxy: {}
replicas: 1
requests:
read: {}
write: {}
storage:
s3:
bucket: image-registry-us-east-1-c92e88cad85b48ec8b312344dff03c82-392c
region: us-east-1
status:
...Edit the config/instance object:
$ oc edit configs.imageregistry.operator.openshift.io/clusterModify the spec section of the object to resemble the following YAML:
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
namespaces:
- openshift-image-registry
topologyKey: kubernetes.io/hostname
weight: 100
logLevel: Normal
managementState: Managed
nodeSelector:
node-role.kubernetes.io/infra: ""Verify the registry pod has been moved to the infrastructure node.
Run the following command to identify the node where the registry pod is located:
$ oc get pods -o wide -n openshift-image-registryConfirm the node has the label you specified:
$ oc describe node <node_name>Review the command output and confirm that node-role.kubernetes.io/infra is in the LABELS list.
By default, the Prometheus Cluster Monitoring stack, which contains Prometheus, Grafana, and AlertManager, is deployed to provide cluster monitoring. It is managed by the Cluster Monitoring Operator. To move its components to different machines, you create and apply a custom config map.
Save the following ConfigMap definition as the cluster-monitoring-configmap.yaml file:
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |+
alertmanagerMain:
nodeSelector:
node-role.kubernetes.io/infra: ""
prometheusK8s:
nodeSelector:
node-role.kubernetes.io/infra: ""
prometheusOperator:
nodeSelector:
node-role.kubernetes.io/infra: ""
grafana:
nodeSelector:
node-role.kubernetes.io/infra: ""
k8sPrometheusAdapter:
nodeSelector:
node-role.kubernetes.io/infra: ""
kubeStateMetrics:
nodeSelector:
node-role.kubernetes.io/infra: ""
telemeterClient:
nodeSelector:
node-role.kubernetes.io/infra: ""
openshiftStateMetrics:
nodeSelector:
node-role.kubernetes.io/infra: ""
thanosQuerier:
nodeSelector:
node-role.kubernetes.io/infra: ""Running this config map forces the components of the monitoring stack to redeploy to infrastructure nodes.
Apply the new config map:
$ oc create -f cluster-monitoring-configmap.yamlWatch the monitoring pods move to the new machines:
$ watch 'oc get pod -n openshift-monitoring -o wide'If a component has not moved to the infra node, delete the pod with this component:
$ oc delete pod -n openshift-monitoring <pod>The component from the deleted pod is re-created on the infra node.
You can configure the Cluster Logging Operator to deploy the pods for logging subsystem components, such as Elasticsearch and Kibana, to different nodes. You cannot move the Cluster Logging Operator pod from its installed location.
For example, you can move the Elasticsearch pods to a separate node because of high CPU, memory, and disk requirements.
The logging subsystem for Red Hat OpenShift and Elasticsearch must be installed. These features are not installed by default.
Edit the ClusterLogging custom resource (CR) in the openshift-logging project:
$ oc edit ClusterLogging instanceapiVersion: logging.openshift.io/v1
kind: ClusterLogging
...
spec:
collection:
logs:
fluentd:
resources: null
type: fluentd
logStore:
elasticsearch:
nodeCount: 3
nodeSelector: (1)
node-role.kubernetes.io/infra: ''
redundancyPolicy: SingleRedundancy
resources:
limits:
cpu: 500m
memory: 16Gi
requests:
cpu: 500m
memory: 16Gi
storage: {}
type: elasticsearch
managementState: Managed
visualization:
kibana:
nodeSelector: (1)
node-role.kubernetes.io/infra: ''
proxy:
resources: null
replicas: 1
resources: null
type: kibana
...| 1 | Add a nodeSelector parameter with the appropriate value to the component you want to move. You can use a nodeSelector in the format shown or use <key>: <value> pairs, based on the value specified for the node. |
To verify that a component has moved, you can use the oc get pod -o wide command.
For example:
You want to move the Kibana pod from the ip-10-0-147-79.us-east-2.compute.internal node:
$ oc get pod kibana-5b8bdf44f9-ccpq9 -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kibana-5b8bdf44f9-ccpq9 2/2 Running 0 27s 10.129.2.18 ip-10-0-147-79.us-east-2.compute.internal <none> <none>You want to move the Kibana pod to the ip-10-0-139-48.us-east-2.compute.internal node, a dedicated infrastructure node:
$ oc get nodesNAME STATUS ROLES AGE VERSION
ip-10-0-133-216.us-east-2.compute.internal Ready master 60m v1.22.1
ip-10-0-139-146.us-east-2.compute.internal Ready master 60m v1.22.1
ip-10-0-139-192.us-east-2.compute.internal Ready worker 51m v1.22.1
ip-10-0-139-241.us-east-2.compute.internal Ready worker 51m v1.22.1
ip-10-0-147-79.us-east-2.compute.internal Ready worker 51m v1.22.1
ip-10-0-152-241.us-east-2.compute.internal Ready master 60m v1.22.1
ip-10-0-139-48.us-east-2.compute.internal Ready infra 51m v1.22.1Note that the node has a node-role.kubernetes.io/infra: '' label:
$ oc get node ip-10-0-139-48.us-east-2.compute.internal -o yamlkind: Node
apiVersion: v1
metadata:
name: ip-10-0-139-48.us-east-2.compute.internal
selfLink: /api/v1/nodes/ip-10-0-139-48.us-east-2.compute.internal
uid: 62038aa9-661f-41d7-ba93-b5f1b6ef8751
resourceVersion: '39083'
creationTimestamp: '2020-04-13T19:07:55Z'
labels:
node-role.kubernetes.io/infra: ''
...To move the Kibana pod, edit the ClusterLogging CR to add a node selector:
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
...
spec:
...
visualization:
kibana:
nodeSelector: (1)
node-role.kubernetes.io/infra: ''
proxy:
resources: null
replicas: 1
resources: null
type: kibana| 1 | Add a node selector to match the label in the node specification. |
After you save the CR, the current Kibana pod is terminated and new pod is deployed:
$ oc get podsNAME READY STATUS RESTARTS AGE
cluster-logging-operator-84d98649c4-zb9g7 1/1 Running 0 29m
elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg 2/2 Running 0 28m
elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj 2/2 Running 0 28m
elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78 2/2 Running 0 28m
fluentd-42dzz 1/1 Running 0 28m
fluentd-d74rq 1/1 Running 0 28m
fluentd-m5vr9 1/1 Running 0 28m
fluentd-nkxl7 1/1 Running 0 28m
fluentd-pdvqb 1/1 Running 0 28m
fluentd-tflh6 1/1 Running 0 28m
kibana-5b8bdf44f9-ccpq9 2/2 Terminating 0 4m11s
kibana-7d85dcffc8-bfpfp 2/2 Running 0 33sThe new pod is on the ip-10-0-139-48.us-east-2.compute.internal node:
$ oc get pod kibana-7d85dcffc8-bfpfp -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kibana-7d85dcffc8-bfpfp 2/2 Running 0 43s 10.131.0.22 ip-10-0-139-48.us-east-2.compute.internal <none> <none>After a few moments, the original Kibana pod is removed.
$ oc get podsNAME READY STATUS RESTARTS AGE
cluster-logging-operator-84d98649c4-zb9g7 1/1 Running 0 30m
elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg 2/2 Running 0 29m
elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj 2/2 Running 0 29m
elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78 2/2 Running 0 29m
fluentd-42dzz 1/1 Running 0 29m
fluentd-d74rq 1/1 Running 0 29m
fluentd-m5vr9 1/1 Running 0 29m
fluentd-nkxl7 1/1 Running 0 29m
fluentd-pdvqb 1/1 Running 0 29m
fluentd-tflh6 1/1 Running 0 29m
kibana-7d85dcffc8-bfpfp 2/2 Running 0 62sThe cluster autoscaler adjusts the size of an OpenShift Container Platform cluster to meet its current deployment needs. It uses declarative, Kubernetes-style arguments to provide infrastructure management that does not rely on objects of a specific cloud provider. The cluster autoscaler has a cluster scope, and is not associated with a particular namespace.
The cluster autoscaler increases the size of the cluster when there are pods that fail to schedule on any of the current worker nodes due to insufficient resources or when another node is necessary to meet deployment needs. The cluster autoscaler does not increase the cluster resources beyond the limits that you specify.
The cluster autoscaler computes the total memory, CPU, and GPU on all nodes the cluster, even though it does not manage the control plane nodes. These values are not single-machine oriented. They are an aggregation of all the resources in the entire cluster. For example, if you set the maximum memory resource limit, the cluster autoscaler includes all the nodes in the cluster when calculating the current memory usage. That calculation is then used to determine if the cluster autoscaler has the capacity to add more worker resources.
|
Ensure that the |
Every 10 seconds, the cluster autoscaler checks which nodes are unnecessary in the cluster and removes them. The cluster autoscaler considers a node for removal if the following conditions apply:
The sum of CPU and memory requests of all pods running on the node is less than 50% of the allocated resources on the node.
The cluster autoscaler can move all pods running on the node to the other nodes.
The cluster autoscaler does not have scale down disabled annotation.
If the following types of pods are present on a node, the cluster autoscaler will not remove the node:
Pods with restrictive pod disruption budgets (PDBs).
Kube-system pods that do not run on the node by default.
Kube-system pods that do not have a PDB or have a PDB that is too restrictive.
Pods that are not backed by a controller object such as a deployment, replica set, or stateful set.
Pods with local storage.
Pods that cannot be moved elsewhere because of a lack of resources, incompatible node selectors or affinity, matching anti-affinity, and so on.
Unless they also have a "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation, pods that have a "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation.
For example, you set the maximum CPU limit to 64 cores and configure the cluster autoscaler to only create machines that have 8 cores each. If your cluster starts with 30 cores, the cluster autoscaler can add up to 4 more nodes with 32 cores, for a total of 62.
If you configure the cluster autoscaler, additional usage restrictions apply:
Do not modify the nodes that are in autoscaled node groups directly. All nodes within the same node group have the same capacity and labels and run the same system pods.
Specify requests for your pods.
If you have to prevent pods from being deleted too quickly, configure appropriate PDBs.
Confirm that your cloud provider quota is large enough to support the maximum node pools that you configure.
Do not run additional node group autoscalers, especially the ones offered by your cloud provider.
The horizontal pod autoscaler (HPA) and the cluster autoscaler modify cluster resources in different ways. The HPA changes the deployment’s or replica set’s number of replicas based on the current CPU load. If the load increases, the HPA creates new replicas, regardless of the amount of resources available to the cluster. If there are not enough resources, the cluster autoscaler adds resources so that the HPA-created pods can run. If the load decreases, the HPA stops some replicas. If this action causes some nodes to be underutilized or completely empty, the cluster autoscaler deletes the unnecessary nodes.
The cluster autoscaler takes pod priorities into account. The Pod Priority and Preemption feature enables scheduling pods based on priorities if the cluster does not have enough resources, but the cluster autoscaler ensures that the cluster has resources to run all pods. To honor the intention of both features, the cluster autoscaler includes a priority cutoff function. You can use this cutoff to schedule "best-effort" pods, which do not cause the cluster autoscaler to increase resources but instead run only when spare resources are available.
Pods with priority lower than the cutoff value do not cause the cluster to scale up or prevent the cluster from scaling down. No new nodes are added to run the pods, and nodes running these pods might be deleted to free resources.
This ClusterAutoscaler resource definition shows the parameters and sample values for the cluster autoscaler.
apiVersion: "autoscaling.openshift.io/v1"
kind: "ClusterAutoscaler"
metadata:
name: "default"
spec:
podPriorityThreshold: -10 (1)
resourceLimits:
maxNodesTotal: 24 (2)
cores:
min: 8 (3)
max: 128 (4)
memory:
min: 4 (5)
max: 256 (6)
gpus:
- type: nvidia.com/gpu (7)
min: 0 (8)
max: 16 (9)
- type: amd.com/gpu (7)
min: 0 (8)
max: 4 (9)
scaleDown: (10)
enabled: true (11)
delayAfterAdd: 10m (12)
delayAfterDelete: 5m (13)
delayAfterFailure: 30s (14)
unneededTime: 5m (15)| 1 | Specify the priority that a pod must exceed to cause the cluster autoscaler to deploy additional nodes. Enter a 32-bit integer value. The podPriorityThreshold value is compared to the value of the PriorityClass that you assign to each pod. |
| 2 | Specify the maximum number of nodes to deploy. This value is the total number of machines that are deployed in your cluster, not just the ones that the autoscaler controls. Ensure that this value is large enough to account for all of your control plane and compute machines and the total number of replicas that you specify in your MachineAutoscaler resources. |
| 3 | Specify the minimum number of cores to deploy in the cluster. |
| 4 | Specify the maximum number of cores to deploy in the cluster. |
| 5 | Specify the minimum amount of memory, in GiB, in the cluster. |
| 6 | Specify the maximum amount of memory, in GiB, in the cluster. |
| 7 | Optionally, specify the type of GPU node to deploy. Only nvidia.com/gpu and amd.com/gpu are valid types. |
| 8 | Specify the minimum number of GPUs to deploy in the cluster. |
| 9 | Specify the maximum number of GPUs to deploy in the cluster. |
| 10 | In this section, you can specify the period to wait for each action by using any valid ParseDuration interval, including ns, us, ms, s, m, and h. |
| 11 | Specify whether the cluster autoscaler can remove unnecessary nodes. |
| 12 | Optionally, specify the period to wait before deleting a node after a node has recently been added. If you do not specify a value, the default value of 10m is used. |
| 13 | Specify the period to wait before deleting a node after a node has recently been deleted. If you do not specify a value, the default value of 10s is used. |
| 14 | Specify the period to wait before deleting a node after a scale down failure occurred. If you do not specify a value, the default value of 3m is used. |
| 15 | Specify the period before an unnecessary node is eligible for deletion. If you do not specify a value, the default value of 10m is used. |
|
When performing a scaling operation, the cluster autoscaler remains within the ranges set in the The minimum and maximum CPUs, memory, and GPU values are determined by calculating those resources on all nodes in the cluster, even if the cluster autoscaler does not manage the nodes. For example, the control plane nodes are considered in the total memory in the cluster, even though the cluster autoscaler does not manage the control plane nodes. |
To deploy the cluster autoscaler, you create an instance of the ClusterAutoscaler resource.
Create a YAML file for the ClusterAutoscaler resource that contains the customized resource definition.
Create the resource in the cluster:
$ oc create -f <filename>.yaml (1)| 1 | <filename> is the name of the resource file that you customized. |
The machine autoscaler adjusts the number of Machines in the machine sets that you deploy in an OpenShift Container Platform cluster. You can scale both the default worker machine set and any other machine sets that you create. The machine autoscaler makes more Machines when the cluster runs out of resources to support more deployments. Any changes to the values in MachineAutoscaler resources, such as the minimum or maximum number of instances, are immediately applied to the machine set they target.
|
You must deploy a machine autoscaler for the cluster autoscaler to scale your machines. The cluster autoscaler uses the annotations on machine sets that the machine autoscaler sets to determine the resources that it can scale. If you define a cluster autoscaler without also defining machine autoscalers, the cluster autoscaler will never scale your cluster. |
This MachineAutoscaler resource definition shows the parameters and sample values for the machine autoscaler.
apiVersion: "autoscaling.openshift.io/v1beta1"
kind: "MachineAutoscaler"
metadata:
name: "worker-us-east-1a" (1)
namespace: "openshift-machine-api"
spec:
minReplicas: 1 (2)
maxReplicas: 12 (3)
scaleTargetRef: (4)
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet (5)
name: worker-us-east-1a (6)| 1 | Specify the machine autoscaler name. To make it easier to identify which machine set this machine autoscaler scales, specify or include the name of the machine set to scale. The machine set name takes the following form: <clusterid>-<machineset>-<aws-region-az>. |
||
| 2 | Specify the minimum number machines of the specified type that must remain in the specified zone after the cluster autoscaler initiates cluster scaling. If running in AWS, GCP, Azure, RHOSP, or vSphere, this value can be set to 0. For other providers, do not set this value to 0.
You can save on costs by setting this value to
|
||
| 3 | Specify the maximum number machines of the specified type that the cluster autoscaler can deploy in the specified AWS zone after it initiates cluster scaling. Ensure that the maxNodesTotal value in the ClusterAutoscaler resource definition is large enough to allow the machine autoscaler to deploy this number of machines. |
||
| 4 | In this section, provide values that describe the existing machine set to scale. | ||
| 5 | The kind parameter value is always MachineSet. |
||
| 6 | The name value must match the name of an existing machine set, as shown in the metadata.name parameter value. |
To deploy the machine autoscaler, you create an instance of the MachineAutoscaler resource.
Create a YAML file for the MachineAutoscaler resource that contains the customized resource definition.
Create the resource in the cluster:
$ oc create -f <filename>.yaml (1)| 1 | <filename> is the name of the resource file that you customized. |
You can turn on a subset of the current Technology Preview features on for all nodes in the cluster by editing the FeatureGate custom resource (CR).
You can use the FeatureGate custom resource (CR) to enable specific feature sets in your cluster. A feature set is a collection of OpenShift Container Platform features that are not enabled by default.
You can activate any of the following feature sets by using the FeatureGate CR:
TechPreviewNoUpgrade. This feature set is a subset of the current Technology Preview features. This feature set allows you to enable these tech preview features on test clusters, where you can fully test them, while leaving the features disabled on production clusters. Enabling this feature set cannot be undone and prevents minor version updates. This feature set is not recommended on production clusters.
The following Technology Preview features are enabled by this feature set:
Azure Disk CSI Driver Operator. Enables the provisioning of persistent volumes (PVs) by using the Container Storage Interface (CSI) driver for Microsoft Azure Disk Storage.
VMware vSphere CSI Driver Operator. Enables the provisioning of persistent volumes (PVs) by using the Container Storage Interface (CSI) VMware vSphere driver for Virtual Machine Disk (VMDK) volumes.
CSI automatic migration. Enables the automatic migration of supported in-tree volume plug-ins to their equivalent Container Storage Interface (CSI) drivers. Available as a Technology Preview for:
Amazon Web Services (AWS) Elastic Block Storage (EBS)
OpenStack Cinder
Azure Disk
Azure File
Google Cloud Platform Persistent Disk (CSI)
VMware vSphere
Cluster Cloud Controller Manager Operator. Enables the Cluster Cloud Controller Manager Operator rather than the in-tree cloud controller. Available as a Technology Preview for:
Amazon Web Services (AWS)
Microsoft Azure
Red Hat OpenStack Platform (RHOSP)
Insights Operator. Enables the importing of RHEL Simple Content Access (SCA) certificates from Red Hat OpenShift Cluster Manager.
You can use the OpenShift CLI (oc) to enable feature sets for all of the nodes in a cluster by editing the FeatureGate custom resource (CR).
You have installed the OpenShift CLI (oc).
To enable feature sets:
Edit the FeatureGate CR named cluster:
$ oc edit featuregate clusterapiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
name: cluster (1)
spec:
featureSet: TechPreviewNoUpgrade (2)| 1 | The name of the FeatureGate CR must be cluster. |
| 2 | Add the feature set that you want to enable:
|
After you save the changes, new machine configs are created, the machine config pools are updated, and scheduling on each node is disabled while the change is being applied.
|
Enabling the |
You can verify that the feature gates are enabled by looking at the kubelet.conf file on a node after the nodes return to the ready state.
Start a debug session for a node:
$ oc debug node/<node_name>Change your root directory to the host:<