Chapter 5. Managing compute nodes using Red Hat build of Karpenter

Procedure

1. Create the trust policy for the Karpenter service account:

   $ cat > trust-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):oidc-provider/$(rosa describe cluster -c $CLUSTER_NAME -o json | jq -r .aws.sts.oidc_endpoint_url | sed 's|https://||')"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringEquals": {
                       "$(rosa describe cluster -c $CLUSTER_NAME -o json | jq -r .aws.sts.oidc_endpoint_url | sed 's|https://||'):sub": "system:serviceaccount:kube-system:karpenter"
                   }
               }
           }
       ]
   }
   EOF

2. Create the IAM role:

   $ aws iam create-role --role-name rosa-karpenter-controller-role-${CLUSTER_NAME} \
     --assume-role-policy-document file://trust-policy.json \
     --tags Key=red-hat-managed,Value=true

3. Attach the AWS managed IAM policy:

   $ aws iam attach-role-policy \
     --role-name rosa-karpenter-controller-role-${CLUSTER_NAME} \
     --policy-arn arn:aws:iam::aws:policy/service-role/ROSAKarpenterControllerPolicy

Procedure

1. Wait for the cluster to become ready:

   $ rosa describe cluster -c $CLUSTER_ID | grep -i State

   Example output

   State:                      ready

2. Ensure that your Red Hat build of Karpenter IAM role is correctly set:

   $ ROLE_ARN=$(aws iam get-role --role-name rosa-karpenter-controller-role-${CLUSTER_NAME} --query 'Role.Arn' --output text)

3. Enable the Red Hat build of Karpenter:

   $ rosa edit cluster -c $CLUSTER_ID \
     --autonode=enabled \
     --autonode-iam-role-arn=$ROLE_ARN

4. If you do not already have cluster admin access, create a cluster admin user:

   $ rosa create admin -c $CLUSTER_ID

5. Log in to the cluster using the credentials from the previous command:

   $ oc login <api_url> --username cluster-admin --password <password>

6. Verify that the Red Hat build of Karpenter custom resource definitions (CRDs) are present:

   $ oc get ec2nodeclass

   Note

   The node pool manifest uses the EC2NodeClass resource.

   Example output

   NAME      READY   AGE
   default   True    5m

   $ oc get openshiftec2nodeclass

   Note

   The OpenshiftEC2NodeClass resource is Red Hat’s wrapper to communicate with the EC2NodeClass resource.

   Example output

   NAME      READY
   default   True

Procedure

1. Export your AWS ID:

   $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

2. Log in to the AWS console.
3. In the AWS console, navigate to IAM > Roles.

4. On your Red Hat build of Karpenter Amazon Resource Name (ARN), update the trust policy to include the following policy specifications:

   Note

   To access this ARN, run:

   $ echo $ROLE_ARN

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "arn:aws:iam::<aws_account_id>:oidc-provider/<oidc-endpoint-URL>"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringEquals": {
                       "<oidc-endpoint-URL>:sub": "system:serviceaccount:kube-system:karpenter"
                   }
               }
           }
       ]
   }

   where:

   <aws_account_id>

   Specifies your AWS Account ID.

   <oidc-endpoint-URL>

   Specifies the OIDC endpoint URL that you acquired.

5. In OpenShift Cluster Manager, select your cluster from the cluster list.
6. On the cluster details screen, select the Edit button next to the status for the Red Hat build of Karpenter.
7. On the Edit Autonode settings dialog box, toggle Enable Autonode.
8. Add your Red Hat build of Karpenter IAM role ARN to the field in this dialog box.
9. Select Save to save your configurations and close the Edit Autonode settings box.

Verification

1. Log in to the cluster:

   $ oc login <api_url> --username cluster-admin --password <password>

2. Verify that the Red Hat build of Karpenter custom resource definitions (CRDs) are present:

   $ oc get ec2nodeclass

   Example output

   NAME      READY   AGE
   default   True    5m

   $ oc get openshiftec2nodeclass

   Example output

   NAME      READY
   default   True

Procedure

1. Create a node pool manifest:

   $ cat > nodepool.yaml <<'EOF'
   apiVersion: karpenter.sh/v1
   kind: NodePool
   metadata:
     name: default-np
   spec:
     template:
       metadata:
         labels:
           autonode: "true"
       spec:
         requirements:
         - key: node.kubernetes.io/instance-type
           operator: In
           values:
           - c5.xlarge
         - key: karpenter.sh/capacity-type
           operator: In
           values: ["on-demand"]
         nodeClassRef:
           group: karpenter.k8s.aws
           kind: EC2NodeClass
           name: default
   EOF

   where:

   nodeClassRef.kind

   Required field that must use the EC2NodeClass type.

   spec.labels

   Optional field that you can use to place pods by using labels.

   Note

   For a list of all requirements available under spec.requirements, see the Additional resources.

2. Apply the node pool:

   $ oc apply -f nodepool.yaml

3. Verify the node pool is ready:

   $ oc get nodepool

   Example output

   NAME         NODECLASS   NODES   READY   AGE
   default-np   default     0       True    3s

Procedure

1. Create a node pool manifest that references the non-default OpenshiftEC2NodeClass:

   $ cat > nodepool.yaml <<'EOF'
   apiVersion: karpenter.sh/v1
   kind: NodePool
   metadata:
     name: custom-np
   spec:
     template:
       spec:
         requirements:
         - key: "karpenter.k8s.aws/instance-cpu"
           operator: Gte
           values: ["4"]
         - key: karpenter.k8s.aws/instance-category
           operator: In
           values: ["c","r","m"]
         nodeClassRef:
           group: karpenter.k8s.aws
           kind: EC2NodeClass
           name: <non-default-nodeclass-name>
   
   EOF

   where:

   nodeClassRef.name

   The name of the non-default OpenshiftEC2NodeClass resource. This value must match the metadata.name of an existing OpenshiftEC2NodeClass.

2. Apply the node pool:

   $ oc apply -f nodepool.yaml

Procedure

1. Create an OpenshiftEC2NodeClass manifest:

   $ cat > openshiftec2nodeclass.yaml <<'EOF'
   apiVersion: karpenter.hypershift.openshift.io/v1
   kind: OpenshiftEC2NodeClass
   metadata:
     name: custom-nodeclass
   spec:
     tags:
       team: "engineering"
   EOF

   For a complete list of configurable fields, see "OpenshiftEC2NodeClass configuration fields".

2. Apply the manifest:

   $ oc apply -f openshiftec2nodeclass.yaml

Verification

1. Verify that the OpenshiftEC2NodeClass resource was created:

   $ oc get openshiftec2nodeclass

   Example output

   NAME               READY   AGE
   default            True    1h
   custom-nodeclass   True    5s

2. Verify the resource is ready by checking its status conditions:

   $ oc get openshiftec2nodeclass/custom-nodeclass -o json | jq '.status.conditions'

   If the Ready condition is False, see the troubleshooting guidance in Additional resources.

Procedure

1. View the current configuration of the OpenshiftEC2NodeClass resource:

   $ oc get openshiftec2nodeclass/<nodeclass_name> -o yaml

2. Edit the resource:

   $ oc edit openshiftec2nodeclass/<nodeclass_name>

   Make the necessary changes to the spec fields. For a complete list of configurable fields, see "OpenshiftEC2NodeClass configuration fields".

3. Save and close the editor to apply the changes.

Verification

1. Verify that the resource is ready after the update:

   $ oc get openshiftec2nodeclass/<nodeclass_name> -o json | jq '.status.conditions[] | select(.type=="Ready")'

   Confirm that the status value is True.

2. If you updated spec.version, verify that the node pool nodes are rolling:

   $ oc get nodes

Procedure

1. Identify all node pools and the OpenshiftEC2NodeClass each one references:

   $ oc get nodepool -o json | jq -r '.items[] | "\(.metadata.name) -> \(.spec.template.spec.nodeClassRef.name)"'

2. For each node pool identified in the previous step, either delete the node pool or update it to reference a different OpenshiftEC2NodeClass:

   • To delete a node pool:

     $ oc delete nodepool <nodepool_name>

   • To update a node pool to reference a different OpenshiftEC2NodeClass:

     $ oc edit nodepool <nodepool_name>

     Change the spec.template.spec.nodeClassRef.name field to the name of a different OpenshiftEC2NodeClass.

3. Delete the OpenshiftEC2NodeClass resource:

   $ oc delete openshiftec2nodeclass <nodeclass_name>

Verification

1. Verify that the resource was deleted:

   $ oc get openshiftec2nodeclass

   Confirm that the deleted resource is no longer listed.

2. Verify that no node pools reference the deleted resource:

   $ oc get nodepool -o json | jq -r '.items[] | select(.spec.template.spec.nodeClassRef.name=="<nodeclass_name>") | .metadata.name'

   This command should return no output.

Procedure

1. Obtain the ARN of the KMS key you want to use. If you want to use the cluster’s default KMS key, run the following command:

   $ rosa describe cluster -c $CLUSTER_NAME -o json | jq -r '.aws.kms_key_arn'

2. Create an OpenshiftEC2NodeClass manifest with the customer-managed key:

   $ cat > openshiftec2nodeclass-cmk.yaml <<'EOF'
   apiVersion: karpenter.hypershift.openshift.io/v1
   kind: OpenshiftEC2NodeClass
   metadata:
     name: cmk-nodeclass
   spec:
     blockDeviceMappings:
       - deviceName: /dev/xvda
         ebs:
           encrypted: Encrypted
           volumeSizeGiB: <root_volume_size>
           kmsKeyID: <kms_key_arn>
   EOF

   where:

   blockDeviceMappings.ebs.encrypted

   Must be set to Encrypted to enable encryption.

   blockDeviceMappings.ebs.volumeSizeGiB

   The size of the root volume in GiB.

   blockDeviceMappings.ebs.kmsKeyID

   The ARN of the symmetric encryption KMS key to use for root volume encryption.

3. Apply the manifest:

   $ oc apply -f openshiftec2nodeclass-cmk.yaml