Cloud Credential Operator

Purpose

The Cloud Credential Operator (CCO) manages cloud provider credentials as Kubernetes custom resource definitions (CRDs). The CCO syncs on credentialsRequest custom resources (CRs) to allow OpenShift Container Platform components to request cloud provider credentials with the specific permissions that are required for the cluster to run.

By setting different values for the credentialsMode parameter in the install-config.yaml file, the CCO can be configured to operate in several different modes. If no mode is specified, or the credentialsMode parameter is set to an empty string (""), the CCO operates in its default mode.

Default behavior

For platforms where multiple modes are supported (AWS, Azure, and GCP), when the CCO operates in its default mode, it checks the provided credentials dynamically to determine for which mode they are sufficient to process credentialsRequest CRs.

By default, the CCO determines whether the credentials are sufficient for mint mode, which is the preferred mode of operation, and uses those credentials to create appropriate credentials for components in the cluster. If the credentials are not sufficient for mint mode, it determines whether they are sufficient for passthrough mode. If the credentials are not sufficient for passthrough mode, the CCO cannot adequately process credentialsRequest CRs.

The CCO cannot verify whether Azure credentials are sufficient for passthrough mode. If Azure credentials are insufficient for mint mode, the CCO operates with the assumption that the credentials are sufficient for passthrough mode.

If the provided credentials are determined to be insufficient during installation, the installation fails. For AWS, the installer fails early in the process and indicates which required permissions are missing. Other providers might not provide specific information about the cause of the error until errors are encountered.

If the credentials are changed after a successful installation and the CCO determines that the new credentials are insufficient, the CCO puts conditions on any new credentialsRequest CRs to indicate that it cannot process them because of the insufficient credentials.

To resolve insufficient credentials issues, provide a credential with sufficient permissions. If an error occurred during installation, try installing again. For issues with new credentialsRequest CRs, wait for the CCO to try to process the CR again. As an alternative, you can manually create IAM for AWS, Azure, or GCP. For details, see the Manually creating IAM section of the installation content for AWS, Azure, or GCP.

Modes

By setting different values for the credentialsMode parameter in the install-config.yaml file, the CCO can be configured to operate in mint, passthrough, or manual mode. These options provide transparency and flexibility in how the CCO uses cloud credentials to process credentialsRequest CRs in the cluster, and allow the CCO to be configured to suit the security requirements of your organization. Not all CCO modes are supported for all cloud providers.

Mint mode

Mint mode is supported for AWS, Azure, and GCP.

Mint mode is the default and recommended best practice setting for the CCO to use. In this mode, the CCO uses the provided admin-level cloud credential to run the cluster.

If the credential is not removed after installation, it is stored and used by the CCO to process credentialsRequest CRs for components in the cluster and create new credentials for each with only the specific permissions that are required. The continuous reconciliation of cloud credentials in mint mode allows actions that require additional credentials or permissions, such as upgrading, to proceed.

The requirement that mint mode stores the admin-level credential in the cluster kube-system namespace might not suit the security requirements of every organization.

When using the CCO in mint mode, ensure that the credential you provide meets the requirements of the cloud on which you are running or installing OpenShift Container Platform. If the provided credentials are not sufficient for mint mode, the CCO cannot create an IAM user.

Table 1. Mint mode credential requirements
Cloud Permissions

AWS

  • iam:CreateAccessKey

  • iam:CreateUser

  • iam:DeleteAccessKey

  • iam:DeleteUser

  • iam:DeleteUserPolicy

  • iam:GetUser

  • iam:GetUserPolicy

  • iam:ListAccessKeys

  • iam:PutUserPolicy

  • iam:TagUser

  • iam:SimulatePrincipalPolicy

Azure

Service principal with the permissions specified in the Creating a service principal section of the Configuring an Azure account content.

GCP

  • resourcemanager.projects.get

  • serviceusage.services.list

  • iam.serviceAccountKeys.create

  • iam.serviceAccountKeys.delete

  • iam.serviceAccounts.create

  • iam.serviceAccounts.delete

  • iam.serviceAccounts.get

  • iam.roles.get

  • resourcemanager.projects.getIamPolicy

  • resourcemanager.projects.setIamPolicy

Mint mode with removal or rotation of the admin-level credential

Mint mode with removal or rotation of the admin-level credential is supported for AWS in OpenShift Container Platform version 4.4 and later.

This option requires the presence of the admin-level credential during installation, but the credential is not stored in the cluster permanently and does not need to be long-lived.

After installing OpenShift Container Platform in mint mode, you can remove the admin-level credential Secret from the cluster. If you remove the Secret, the CCO uses a previously minted read-only credential that allows it to verify whether all credentialsRequest CRs have their required permissions. Once removed, the associated credential can be destroyed on the underlying cloud if desired.

The admin-level credential is not required unless something that requires an admin-level credential needs to be changed, for instance during an upgrade. Prior to each upgrade, you must reinstate the credential Secret with the admin-level credential. If the credential is not present, the upgrade might be blocked.

Passthrough mode

Passthrough mode is supported for AWS, Azure, GCP, Red Hat OpenStack Platform (RHOSP), Red Hat Virtualization (RHV), and VMware vSphere.

In passthrough mode, the CCO passes the provided cloud credential to the components that request cloud credentials. The credential must have permissions to perform the installation and complete the operations that are required by components in the cluster, but does not need to be able to create new credentials. The CCO does not attempt to create additional limited-scoped credentials in passthrough mode.

Passthrough mode permissions requirements

When using the CCO in passthrough mode, ensure that the credential you provide meets the requirements of the cloud on which you are running or installing OpenShift Container Platform. If the provided credentials the CCO passes to a component that creates a credentialsRequest CR are not sufficient, that component will report an error when it tries to call an API that it does not have permissions for.

The credential you provide for passthrough mode in AWS, Azure, or GCP must have all the requested permissions for all credentialsRequest CRs that are required by the version of OpenShift Container Platform you are running or installing. To locate the credentialsRequest CRs that are required for your cloud provider, see the Manually creating IAM section of the installation content for AWS, Azure, or GCP.

To install an OpenShift Container Platform cluster on Red Hat OpenStack Platform (RHOSP), the CCO requires a credential with the permissions of a member user role.

To install an OpenShift Container Platform cluster on Red Hat Virtualization (RHV), the CCO requires a credential with the following privileges:

  • DiskOperator

  • DiskCreator

  • UserTemplateBasedVm

  • TemplateOwner

  • TemplateCreator

  • ClusterAdmin on the specific cluster that is targeted for OpenShift Container Platform deployment

To install an OpenShift Container Platform cluster on VMware vSphere, the CCO requires a credential with the following vSphere privileges:

Table 2. Required vSphere privileges
Category Privileges

Datastore

Allocate space

Folder

Create folder, Delete folder

vSphere Tagging

All privileges

Network

Assign network

Resource

Assign virtual machine to resource pool

Profile-driven storage

All privileges

vApp

All privileges

Virtual machine

All privileges

Passthrough mode credential maintenance

If credentialsRequest CRs change over time as the cluster is upgraded, you must manually update the passthrough mode credential to meet the requirements. To avoid credentials issues during an upgrade, check the credentialsRequest CRs in the release image for the new version of OpenShift Container Platform before upgrading. To locate the credentialsRequest CRs that are required for your cloud provider, see the Manually creating IAM section of the installation content for AWS, Azure, or GCP.

Reducing permissions after installation

When using passthrough mode, each component has the same permissions used by all other components. If you do not reduce the permissions after installing, all components have the broad permissions that are required to run the installer.

After installation, you can reduce the permissions on your credential to only those that are required to run the cluster, as defined by the credentialsRequest CRs in the release image for the version of OpenShift Container Platform that you are using.

To locate the credentialsRequest CRs that are required for AWS, Azure, or GCP and learn how to change the permissions the CCO uses, see the Manually creating IAM section of the installation content for AWS, Azure, or GCP.

Manual mode

Manual mode is supported for AWS.

In manual mode, a user manages cloud credentials instead of the CCO. To use this mode, you must examine the credentialsRequest CRs in the release image for the version of OpenShift Container Platform that you are running or installing, create corresponding credentials in the underlying cloud provider, and create Kubernetes Secrets in the correct namespaces to satisfy all credentialsRequest CRs for the cluster’s cloud provider.

Using manual mode allows each cluster component to have only the permissions it requires, without storing an admin-level credential in the cluster. This mode also does not require connectivity to the AWS public IAM endpoint. However, you must manually reconcile permissions with new release images for every upgrade.

For information about configuring AWS to use manual mode, see Manually creating IAM for AWS.

Disabled CCO

Disabled CCO is supported for Azure and GCP.

To manually manage credentials for Azure or GCP, you must disable the CCO. Disabling the CCO has many of the same configuration and maintenance requirements as running the CCO in manual mode, but is accomplished by a different process. For more information, see the Manually creating IAM section of the installation content for Azure or GCP.

Project

CRDs

  • credentialsrequests.cloudcredential.openshift.io

    • Scope: Namespaced

    • CR: credentialsrequest

    • Validation: Yes

Configuration objects

No configuration required.

Cluster Authentication Operator

Purpose

The Cluster Authentication Operator installs and maintains the Authentication Custom Resource in a cluster and can be viewed with:

$ oc get clusteroperator authentication -o yaml

Project

Cluster Autoscaler Operator

Purpose

The Cluster Autoscaler Operator manages deployments of the OpenShift Cluster Autoscaler using the cluster-api provider.

Project

CRDs

  • ClusterAutoscaler: This is a singleton resource, which controls the configuration of the cluster’s autoscaler instance. The Operator will only respond to the ClusterAutoscaler resource named default in the managed namespace, the value of the WATCH_NAMESPACE environment variable.

  • MachineAutoscaler: This resource targets a node group and manages the annotations to enable and configure autoscaling for that group, the min and max size. Currently only MachineSet objects can be targeted.

Cluster Image Registry Operator

Purpose

The Cluster Image Registry Operator manages a singleton instance of the OpenShift Container Platform registry. It manages all configuration of the registry, including creating storage.

On initial start up, the Operator creates a default image-registry resource instance based on the configuration detected in the cluster. This indicates what cloud storage type to use based on the cloud provider.

If insufficient information is available to define a complete image-registry resource, then an incomplete resource is defined and the Operator updates the resource status with information about what is missing.

The Cluster Image Registry Operator runs in the openshift-image-registry namespace and it also manages the registry instance in that location. All configuration and workload resources for the registry reside in that namespace.

Project

Cluster Monitoring Operator

Purpose

The Cluster Monitoring Operator manages and updates the Prometheus-based cluster monitoring stack deployed on top of OpenShift Container Platform.

Project

CRDs

  • alertmanagers.monitoring.coreos.com

    • Scope: Namespaced

    • CR: alertmanager

    • Validation: Yes

  • prometheuses.monitoring.coreos.com

    • Scope: Namespaced

    • CR: prometheus

    • Validation: Yes

  • prometheusrules.monitoring.coreos.com

    • Scope: Namespaced

    • CR: prometheusrule

    • Validation: Yes

  • servicemonitors.monitoring.coreos.com

    • Scope: Namespaced

    • CR: servicemonitor

    • Validation: Yes

Configuration objects

$ oc -n openshift-monitoring edit cm cluster-monitoring-config

Cluster Network Operator

Purpose

The Cluster Network Operator installs and upgrades the networking components on an OpenShift Kubernetes cluster.

OpenShift Controller Manager Operator

Purpose

The OpenShift Controller Manager Operator installs and maintains the OpenShiftControllerManager Custom Resource in a cluster and can be viewed with:

$ oc get clusteroperator openshift-controller-manager -o yaml

The Custom Resource Definition openshiftcontrollermanagers.operator.openshift.io can be viewed in a cluster with:

$ oc get crd openshiftcontrollermanagers.operator.openshift.io -o yaml

Project

Cluster Samples Operator

Purpose

The Cluster Samples Operator manages the sample imagestreams and templates stored in the openshift namespace.

On initial start up, the Operator creates the default samples configuration resource to initiate the creation of the imagestreams and templates. The configuration object is a cluster scoped object with the key cluster and type configs.samples.

The imagestreams are the Red Hat Enterprise Linux CoreOS (RHCOS)-based OpenShift Container Platform imagestreams pointing to images on registry.redhat.io. Similarly, the templates are those categorized as OpenShift Container Platform templates.

The Cluster Samples Operator deployment is contained within the openshift-cluster-samples-operator namespace. On start up, the install pull secret is used by the imagestream import logic in the internal registry and API server to authenticate with registry.redhat.io. An administrator can create any additional secrets in the openshift namespace if they change the registry used for the sample imagestreams. If created, those secrets contain the content of a Docker config.json needed to facilitate image import.

The image for the Cluster Samples Operator contains imagestream and template definitions for the associated OpenShift Container Platform release. After the Cluster Samples Operator creates a sample, it adds an annotation that denotes the OpenShift Container Platform version that it is compatible with. The Operator uses this annotation to ensure that each sample matches it’s release version. Samples outside of its inventory are ignored, as are skipped samples. Modifications to any samples that are managed by the Operator are allowed as long as the version annotation is not modified or deleted. However, on an upgrade, as the version annotation will change, those modifications can get replaced as the sample will be updated with the newer version. The Jenkins images are part of the image payload from the installation and are tagged into the imagestreams directly.

The samples resource includes a finalizer, which cleans up the following upon its deletion:

  • Operator-managed imagestreams

  • Operator-managed templates

  • Operator-generated configuration resources

  • Cluster status resources

Upon deletion of the samples resource, the Cluster Samples Operator recreates the resource using the default configuration.

Project

Cluster Storage Operator

Purpose

The Cluster Storage Operator sets OpenShift Container Platform cluster-wide storage defaults. It ensures a default storage class exists for OpenShift Container Platform clusters.

Project

Configuration

No configuration is required.

Notes

  • The Cluster Storage Operator supports Amazon Web Services (AWS) and Red Hat OpenStack Platform (RHOSP).

  • The created storage class can be made non-default by editing its annotation, but the storage class cannot be deleted as long as the Operator runs.

Cluster Version Operator

Purpose

Project

Console Operator

Purpose

The Console Operator installs and maintains the OpenShift Container Platform web console on a cluster.

Project

DNS Operator

Purpose

The DNS Operator deploys and manages CoreDNS to provide a name resolution service to pods that enables DNS-based Kubernetes Service discovery in OpenShift Container Platform.

The Operator creates a working default deployment based on the cluster’s configuration.

  • The default cluster domain is cluster.local.

  • Configuration of the CoreDNS Corefile or Kubernetes plug-in is not yet supported.

The DNS Operator manages CoreDNS as a Kubernetes daemon set exposed as a service with a static IP. CoreDNS runs on all nodes in the cluster.

Project

etcd cluster Operator

Purpose

The etcd cluster Operator automates etcd cluster scaling, enables etcd monitoring and metrics, and simplifies disaster recovery procedures.

Project

CRDs

  • etcds.operator.openshift.io

    • Scope: Cluster

    • CR: etcd

    • Validation: Yes

Configuration objects

$ oc edit etcd cluster

Ingress Operator

Purpose

The Ingress Operator configures and manages the OpenShift Container Platform router.

Project

CRDs

  • clusteringresses.ingress.openshift.io

    • Scope: Namespaced

    • CR: clusteringresses

    • Validation: No

Configuration objects

  • Cluster config

    • Type Name: clusteringresses.ingress.openshift.io

    • Instance Name: default

    • View Command:

      $ oc get clusteringresses.ingress.openshift.io -n openshift-ingress-operator default -o yaml

Notes

The Ingress Operator sets up the router in the openshift-ingress project and creates the deployment for the router:

$ oc get deployment -n openshift-ingress

The Ingress Operator uses the clusterNetwork[].cidr from the network/cluster status to determine what mode (IPv4, IPv6, or dual stack) the managed ingress controller (router) should operate in. For example, if clusterNetwork contains only a v6 cidr, then the ingress controller will operate in v6-only mode. In the following example, ingress controllers managed by the Ingress Operator will run in v4-only mode because only one cluster network exists and the network is a v4 cidr:

$ oc get network/cluster -o jsonpath='{.status.clusterNetwork[*]}'
Example output
map[cidr:10.128.0.0/14 hostPrefix:23]

Kubernetes API Server Operator

Purpose

The Kubernetes API Server Operator manages and updates the Kubernetes API server deployed on top of OpenShift Container Platform. The Operator is based on the OpenShift library-go framework and it is installed using the Cluster Version Operator (CVO).

Project

CRDs

  • kubeapiservers.operator.openshift.io

    • Scope: Cluser

    • CR: kubeapiserver

    • Validation: Yes

Configuration objects

$ oc edit kubeapiserver

Kubernetes Controller Manager Operator

Purpose

The Kubernetes Controller Manager Operator manages and updates the Kubernetes Controller Manager deployed on top of OpenShift Container Platform. The Operator is based on OpenShift library-go framework and it is installed via the Cluster Version Operator (CVO).

It contains the following components:

  • Operator

  • Bootstrap manifest renderer

  • Installer based on static pods

  • Configuration observer

By default, the Operator exposes Prometheus metrics through the metrics service.

Project

Kubernetes Scheduler Operator

Purpose

The Kubernetes Scheduler Operator manages and updates the Kubernetes Scheduler deployed on top of OpenShift Container Platform. The operator is based on the OpenShift Container Platform library-go framework and it is installed with the Cluster Version Operator (CVO).

The Kubernetes Scheduler Operator contains the following components:

  • Operator

  • Bootstrap manifest renderer

  • Installer based on static pods

  • Configuration observer

By default, the Operator exposes Prometheus metrics through the metrics service.

Project

Configuration

The configuration for the Kubernetes Scheduler is the result of merging:

  • a default configuration.

  • an observed configuration from the spec schedulers.config.openshift.io.

All of these are sparse configurations, invalidated JSON snippets which are merged in order to form a valid configuration at the end.

Machine API Operator

Purpose

The Machine API Operator manages the lifecycle of specific purpose CRDs, controllers, and RBAC objects that extend the Kubernetes API. This declares the desired state of machines in a cluster.

Project

CRDs

  • MachineSet

  • Machine

  • MachineHealthCheck

Machine Config Operator

Purpose

The Machine Config Operator manages and applies configuration and updates of the base operating system and container runtime, including everything between the kernel and kubelet.

There are four components:

  • machine-config-server: Provides Ignition configuration to new machines joining the cluster.

  • machine-config-controller: Coordinates the upgrade of machines to the desired configurations defined by a MachineConfig object. Options are provided to control the upgrade for sets of machines individually.

  • machine-config-daemon: Applies new machine configuration during update. Validates and verifies the machine’s state to the requested machine configuration.

  • machine-config: Provides a complete source of machine configuration at installation, first start up, and updates for a machine.

Project

Marketplace Operator

Purpose

The Marketplace Operator is a conduit to bring off-cluster Operators to your cluster.

Project

Node Tuning Operator

Purpose

The Node Tuning Operator helps you manage node-level tuning by orchestrating the Tuned daemon. The majority of high-performance applications require some level of kernel tuning. The Node Tuning Operator provides a unified management interface to users of node-level sysctls and more flexibility to add custom tuning specified by user needs. The Operator manages the containerized Tuned daemon for OpenShift Container Platform as a Kubernetes daemon set. It ensures the custom tuning specification is passed to all containerized Tuned daemons running in the cluster in the format that the daemons understand. The daemons run on all nodes in the cluster, one per node.

Node-level settings applied by the containerized Tuned daemon are rolled back on an event that triggers a profile change or when the containerized Tuned daemon is terminated gracefully by receiving and handling a termination signal.

The Node Tuning Operator is part of a standard OpenShift Container Platform installation in version 4.1 and later.

Project

Operator Lifecycle Manager Operators

Purpose

Operator Lifecycle Manager (OLM) helps users install, update, and manage the lifecycle of all Operators and their associated services running across their OpenShift Container Platform clusters. It is part of the Operator Framework, an open source toolkit designed to manage Kubernetes native applications (Operators) in an effective, automated, and scalable way.

olm workflow
Figure 1. Operator Lifecycle Manager workflow

OLM runs by default in OpenShift Container Platform 4.6, which aids cluster administrators in installing, upgrading, and granting access to Operators running on their cluster. The OpenShift Container Platform web console provides management screens for cluster administrators to install Operators, as well as grant specific projects access to use the catalog of Operators available on the cluster.

For developers, a self-service experience allows provisioning and configuring instances of databases, monitoring, and big data services without having to be subject matter experts, because the Operator has that knowledge baked into it.

CRDs

Operator Lifecycle Manager (OLM) is composed of two Operators: the OLM Operator and the Catalog Operator.

Each of these Operators is responsible for managing the Custom Resource Definitions (CRDs) that are the basis for the OLM framework:

Table 3. CRDs managed by OLM and Catalog Operators
Resource Short name Owner Description

ClusterServiceVersion

csv

OLM

Application metadata: name, version, icon, required resources, installation, and so on.

InstallPlan

ip

Catalog

Calculated list of resources to be created to automatically install or upgrade a CSV.

CatalogSource

catsrc

Catalog

A repository of CSVs, CRDs, and packages that define an application.

Subscription

sub

Catalog

Used to keep CSVs up to date by tracking a channel in a package.

OperatorGroup

og

OLM

Configures all Operators deployed in the same namespace as the OperatorGroup object to watch for their custom resource (CR) in a list of namespaces or cluster-wide.

Each of these Operators is also responsible for creating resources:

Table 4. Resources created by OLM and Catalog Operators
Resource Owner

Deployments

OLM

ServiceAccounts

(Cluster)Roles

(Cluster)RoleBindings

Custom Resource Definitions (CRDs)

Catalog

ClusterServiceVersions (CSVs)

OLM Operator

The OLM Operator is responsible for deploying applications defined by CSV resources after the required resources specified in the CSV are present in the cluster.

The OLM Operator is not concerned with the creation of the required resources; you can choose to manually create these resources using the CLI or using the Catalog Operator. This separation of concern allows users incremental buy-in in terms of how much of the OLM framework they choose to leverage for their application.

The OLM Operator uses the following workflow:

  1. Watch for ClusterServiceVersion (CSVs) in a namespace and check that requirements are met.

  2. If requirements are met, run the install strategy for the CSV.

    A CSV must be an active member of an OperatorGroup for the install strategy to run.

Catalog Operator

The Catalog Operator is responsible for resolving and installing CSVs and the required resources they specify. It is also responsible for watching CatalogSources for updates to packages in channels and upgrading them, automatically if desired, to the latest available versions.

To track a package in a channel, you can create a Subscription resource configuring the desired package, channel, and the CatalogSource you want to pull updates from. When updates are found, an appropriate InstallPlan is written into the namespace on behalf of the user.

The Catalog Operator uses the following workflow:

  1. Connect to each CatalogSource in the cluster.

  2. Watch for unresolved InstallPlans created by a user, and if found:

    1. Find the CSV matching the name requested and add the CSV as a resolved resource.

    2. For each managed or required CRD, add the CRD as a resolved resource.

    3. For each required CRD, find the CSV that manages it.

  3. Watch for resolved InstallPlans and create all of the discovered resources for it, if approved by a user or automatically.

  4. Watch for CatalogSources and Subscriptions and create InstallPlans based on them.

Catalog Registry

The Catalog Registry stores CSVs and CRDs for creation in a cluster and stores metadata about packages and channels.

A package manifest is an entry in the Catalog Registry that associates a package identity with sets of CSVs. Within a package, channels point to a particular CSV. Because CSVs explicitly reference the CSV that they replace, a package manifest provides the Catalog Operator with all of the information that is required to update a CSV to the latest version in a channel, stepping through each intermediate version.

Additional resources

For more information, see the sections on understanding Operator Lifecycle Manager (OLM).

OpenShift API Server Operator

Purpose

The OpenShift API Server Operator installs and maintains the openshift-apiserver on a cluster.

Project

CRDs

  • openshiftapiservers.operator.openshift.io

    • Scope: Cluster

    • CR: openshiftapiserver

    • Validation: Yes

Prometheus Operator

Purpose

The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances.

Once installed, the Prometheus Operator provides the following features:

  • Create and Destroy: Easily launch a Prometheus instance for your Kubernetes namespace, a specific application or team easily using the Operator.

  • Simple Configuration: Configure the fundamentals of Prometheus like versions, persistence, retention policies, and replicas from a native Kubernetes resource.

  • Target Services via Labels: Automatically generate monitoring target configurations based on familiar Kubernetes label queries; no need to learn a Prometheus specific configuration language.

Project