Managing machines with the Cluster API | Machine management

Cluster API architecture
- The Cluster CAPI Operator
- Primary resources
Sample YAML files
Creating a Cluster API machine set
Troubleshooting clusters that use the Cluster API
- CLI commands return Cluster API machines

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The Cluster API is an upstream project that is integrated into OpenShift Container Platform as a Technology Preview for Amazon Web Services (AWS) and Google Cloud Platform (GCP) clusters. You can use the Cluster API to create and manage machine sets and machines in your OpenShift Container Platform cluster. This capability is in addition or an alternative to managing machines with the Machine API.

For OpenShift Container Platform 4.11 clusters, you can use the Cluster API to perform node host provisioning management actions after the cluster installation finishes. This system enables an elastic, dynamic provisioning method on top of public or private cloud infrastructure.

With the Cluster API Technology Preview, you can create compute machines and machine sets on OpenShift Container Platform clusters for supported providers. You can also explore the features that are enabled by this implementation that might not be available with the Machine API.

Benefits

By using the Cluster API, OpenShift Container Platform users and developers are able to realize the following advantages:

The option to use upstream community Cluster API infrastructure providers which might not be supported by the Machine API.
The opportunity to collaborate with third parties who maintain machine controllers for infrastructure providers.
The ability to use the same set of Kubernetes tools for infrastructure management in OpenShift Container Platform.
The ability to create machine sets using the Cluster API that support features that are not available with the Machine API.

Limitations

Using the Cluster API to manage machines is a Technology Preview feature and has the following limitations:

Only AWS and GCP clusters are supported.
To use this feature, you must enable the TechPreviewNoUpgrade feature set. Enabling this feature set cannot be undone and prevents minor version updates.
You must create the primary resources that the Cluster API requires manually.
Control plane machines cannot be managed by the Cluster API.
Migration of existing machine sets created by the Machine API to Cluster API machine sets is not supported.
Full feature parity with the Machine API is not available.

Cluster API architecture

The OpenShift Container Platform integration of the upstream Cluster API is implemented and managed by the Cluster CAPI Operator. The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, in contrast to the Machine API, which uses the openshift-machine-api namespace.

The Cluster CAPI Operator

The Cluster CAPI Operator is an OpenShift Container Platform Operator that maintains the lifecycle of Cluster API resources. This Operator is responsible for all administrative tasks related to deploying the Cluster API project within an OpenShift Container Platform cluster.

If a cluster is configured correctly to allow the use of the Cluster API, the Cluster CAPI Operator installs the Cluster API Operator on the cluster.

The Cluster CAPI Operator is distinct from the upstream Cluster API Operator.

For more information, see the entry for the Cluster CAPI Operator in the Cluster Operators reference content.

Primary resources

The Cluster API is comprised of the following primary resources. For the Technology Preview of this feature, you must create these resources manually in the openshift-cluster-api namespace.

Cluster: A fundamental unit that represents a cluster that is managed by the Cluster API.
Infrastructure: A provider-specific resource that defines properties that are shared by all the machine sets in the cluster, such as the region and subnets.
Machine template: A provider-specific template that defines the properties of the machines that a machine set creates.
Machine set: A group of machines.

Machine sets are to machines as replica sets are to pods. If you need more machines or must scale them down, you change the replicas field on the machine set to meet your compute needs.

With the Cluster API, a machine set references a Cluster object and a provider-specific machine template.
Machine: A fundamental unit that describes the host for a node.

The Cluster API creates machines based on the configuration in the machine template.

Additional resources

Cluster CAPI Operator

Sample YAML files

For the Cluster API Technology Preview, you must create the primary resources that the Cluster API requires manually. The example YAML files in this section demonstrate how to make these resources work together and configure settings for the machines that they create that are appropriate for your environment.

Sample YAML for a Cluster API cluster resource

The cluster resource defines the name and infrastructure provider for the cluster and is managed by the Cluster API. This resource has the same structure for all providers.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <cluster_name> (1)
  namespace: openshift-cluster-api
spec:
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: <infrastructure_kind> (2)
    name: <cluster_name> (1)
    namespace: openshift-cluster-api

1	Specify the name of the cluster.
2	Specify the infrastructure kind for the cluster. Valid values are: `AWSCluster`: The cluster is running on Amazon Web Services (AWS). `GCPCluster`: The cluster is running on Google Cloud Platform (GCP).

The remaining Cluster API resources are provider-specific. Refer to the example YAML files for your cluster:

Sample YAML files for configuring Amazon Web Services clusters

Some Cluster API resources are provider-specific. The example YAML files in this section show configurations for an Amazon Web Services (AWS) cluster.

Sample YAML for a Cluster API infrastructure resource on Amazon Web Services

The infrastructure resource is provider-specific and defines properties that are shared by all the machine sets in the cluster, such as the region and subnets. The machine set references this resource when creating machines.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster (1)
metadata:
  name: <cluster_name> (2)
  namespace: openshift-cluster-api
spec:
  region: <region> (3)

1	Specify the infrastructure kind for the cluster. This value must match the value for your platform.
2	Specify the name of the cluster.
3	Specify the AWS region.

Sample YAML for a Cluster API machine template resource on Amazon Web Services

The machine template resource is provider-specific and defines the basic properties of the machines that a machine set creates. The machine set references this template when creating machines.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: AWSMachineTemplate (1)
metadata:
  name: <template_name> (2)
  namespace: openshift-cluster-api
spec:
  template:
    spec: (3)
      uncompressedUserData: true
      iamInstanceProfile: ....
      instanceType: m5.large
      cloudInit:
        insecureSkipSecretsManager: true
      ami:
        id: ....
      subnet:
        filters:
        - name: tag:Name
          values:
          - ...
      additionalSecurityGroups:
      - filters:
        - name: tag:Name
          values:
          - ...

1	Specify the machine template kind. This value must match the value for your platform.
2	Specify a name for the machine template.
3	Specify the details for your environment. The values here are examples.

Sample YAML for a Cluster API machine set resource on Amazon Web Services

The machine set resource defines additional properties of the machines that it creates. The machine set also references the infrastructure resource and machine template when creating machines.

apiVersion: cluster.x-k8s.io/v1alpha4
kind: MachineSet
metadata:
  name: <machine_set_name> (1)
  namespace: openshift-cluster-api
spec:
  clusterName: <cluster_name> (2)
  replicas: 1
  selector:
    matchLabels:
      test: example
  template:
    metadata:
      labels:
        test: example
    spec:
      bootstrap:
         dataSecretName: worker-user-data (3)
      clusterName: <cluster_name> (2)
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
        kind: AWSMachineTemplate (4)
        name: <cluster_name> (2)

1	Specify a name for the machine set.
2	Specify the name of the cluster.
3	For the Cluster API Technology Preview, the Operator can use the worker user data secret from `openshift-machine-api` namespace.
4	Specify the machine template kind. This value must match the value for your platform.

Sample YAML files for configuring Google Cloud Platform clusters

Some Cluster API resources are provider-specific. The example YAML files in this section show configurations for a Google Cloud Platform (GCP) cluster.

Sample YAML for a Cluster API infrastructure resource on Google Cloud Platform

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPCluster (1)
metadata:
  name: <cluster_name> (2)
spec:
  network:
    name: <cluster_name>-network (2)
  project: <project> (3)
  region: <region> (4)

1	Specify the infrastructure kind for the cluster. This value must match the value for your platform.
2	Specify the name of the cluster.
3	Specify the GCP project name.
4	Specify the GCP region.

Sample YAML for a Cluster API machine template resource on Google Cloud Platform

The machine template resource is provider-specific and defines the basic properties of the machines that a machine set creates. The machine set references this template when creating machines.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPMachineTemplate (1)
metadata:
  name: <template_name> (2)
  namespace: openshift-cluster-api
spec:
  template:
    spec: (3)
      rootDeviceType: pd-ssd
      rootDeviceSize: 128
      instanceType: n1-standard-4
      image: projects/rhcos-cloud/global/images/rhcos-411-85-202203181601-0-gcp-x86-64
      subnet: <cluster_name>-worker-subnet
      serviceAccounts:
        email: <service_account_email_address>
        scopes:
          - https://www.googleapis.com/auth/cloud-platform
      additionalLabels:
        kubernetes-io-cluster-<cluster_name>: owned
      additionalNetworkTags:
        - <cluster_name>-worker
      ipForwarding: Disabled

1	Specify the machine template kind. This value must match the value for your platform.
2	Specify a name for the machine template.
3	Specify the details for your environment. The values here are examples.

Sample YAML for a Cluster API machine set resource on Google Cloud Platform

The machine set resource defines additional properties of the machines that it creates. The machine set also references the infrastructure resource and machine template when creating machines.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  name: <machine_set_name> (1)
  namespace: openshift-cluster-api
spec:
  clusterName: <cluster_name> (2)
  replicas: 1
  selector:
    matchLabels:
      test: test
  template:
    metadata:
      labels:
        test: test
    spec:
      bootstrap:
         dataSecretName: worker-user-data (3)
      clusterName: <cluster_name> (2)
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: GCPMachineTemplate (4)
        name: <machine_set_name> (1)
      failureDomain: <failure_domain> (5)

1	Specify a name for the machine set.
2	Specify the name of the cluster.
3	For the Cluster API Technology Preview, the Operator can use the worker user data secret from `openshift-machine-api` namespace.
4	Specify the machine template kind. This value must match the value for your platform.
5	Specify the failure domain within the GCP region.

Creating a Cluster API machine set

You can create machine sets that use the Cluster API to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

Deploy an OpenShift Container Platform cluster.
Enable the use of the Cluster API.
Install the OpenShift CLI (oc).
Log in to oc as a user with cluster-admin permission.

Procedure

Create a YAML file that contains the cluster custom resource (CR) and is named <cluster_resource_file>.yaml.

If you are not sure which value to set for the <cluster_name> parameter, you can check the value for an existing Machine API machine set in your cluster.

To list the Machine API machine sets, run the following command:

$ oc get machinesets -n openshift-machine-api (1)

1	Specify the `openshift-machine-api` namespace.

Example output

NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
agl030519-vplxk-worker-us-east-1d   0         0                             55m
agl030519-vplxk-worker-us-east-1e   0         0                             55m
agl030519-vplxk-worker-us-east-1f   0         0                             55m

To display the contents of a specific machine set CR, run the following command:

$ oc get machineset <machineset_name> \
-n openshift-machine-api \
-o yaml

Example output

...
template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: agl030519-vplxk (1)
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: agl030519-vplxk-worker-us-east-1a
...

1	The cluster ID, which you use for the `<cluster_name>` parameter.

Create the cluster CR by running the following command:
```
$ oc create -f <cluster_resource_file>.yaml
```
Verification

To confirm that the cluster CR is created, run the following command:
```
$ oc get cluster
```
Example output
```
NAME           PHASE        AGE  VERSION
<cluster_name> Provisioning 4h6m
```
Create a YAML file that contains the infrastructure CR and is named <infrastructure_resource_file>.yaml.
Create the infrastructure CR by running the following command:
```
$ oc create -f <infrastructure_resource_file>.yaml
```
Verification

To confirm that the infrastructure CR is created, run the following command:
```
$ oc get <infrastructure_kind>
```
where <infrastructure_kind> is the value that corresponds to your platform.
Example output
```
NAME           CLUSTER        READY VPC BASTION IP
<cluster_name> <cluster_name> true
```
Create a YAML file that contains the machine template CR and is named <machine_template_resource_file>.yaml.
Create the machine template CR by running the following command:
```
$ oc create -f <machine_template_resource_file>.yaml
```
Verification

To confirm that the machine template CR is created, run the following command:
```
$ oc get <machine_template_kind>
```
where <machine_template_kind> is the value that corresponds to your platform.
Example output
```
NAME            AGE
<template_name> 77m
```
Create a YAML file that contains the machine set CR and is named <machine_set_resource_file>.yaml.
Create the machine set CR by running the following command:
```
$ oc create -f <machine_set_resource_file>.yaml
```
Verification

To confirm that the machine set CR is created, run the following command:
```
$ oc get machineset -n openshift-cluster-api (1)
```
1 Specify the openshift-cluster-api namespace.
Example output
```
NAME               CLUSTER        REPLICAS READY AVAILABLE AGE VERSION
<machine_set_name> <cluster_name> 1        1     1         17m
```
When the new machine set is available, the REPLICAS and AVAILABLE values match. If the machine set is not available, wait a few minutes and run the command again.

Verification

To verify that the machine set is creating machines according to your desired configuration, you can review the lists of machines and nodes in the cluster.

To view the list of Cluster API machines, run the following command:

$ oc get machine -n openshift-cluster-api (1)

1	Specify the `openshift-cluster-api` namespace.

Example output

NAME                           CLUSTER        NODENAME                               PROVIDERID    PHASE   AGE   VERSION
<machine_set_name>-<string_id> <cluster_name> <ip_address>.<region>.compute.internal <provider_id> Running 8m23s

To view the list of nodes, run the following command:

$ oc get node

Example output

NAME                                     STATUS ROLES  AGE   VERSION
<ip_address_1>.<region>.compute.internal Ready  worker 5h14m v1.24.0+284d62a
<ip_address_2>.<region>.compute.internal Ready  master 5h19m v1.24.0+284d62a
<ip_address_3>.<region>.compute.internal Ready  worker 7m    v1.24.0+284d62a

Troubleshooting clusters that use the Cluster API

Use the information in this section to understand and recover from issues you might encounter. Generally, troubleshooting steps for problems with the Cluster API are similar to those steps for problems with the Machine API.

The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, whereas the Machine API uses the openshift-machine-api namespace. When using oc commands that reference a namespace, be sure to reference the correct one.

CLI commands return Cluster API machines

For clusters that use the Cluster API, oc commands such as oc get machine return results for Cluster API machines. Because the letter c precedes the letter m alphabetically, Cluster API machines appear in the return before Machine API machines do.

To list only Machine API machines, use the fully qualified name machines.machine.openshift.io when running the oc get machine command:
```
$ oc get machines.machine.openshift.io
```
To list only Cluster API machines, use the fully qualified name machines.cluster.x-k8s.io when running the oc get machine command:
```
$ oc get machines.cluster.x-k8s.io
```