SupportConsoleDevelopersStart a trialContact

Chapter 10. Configure AWS Spot instances for Red Hat OpenShift Service on AWS using the CLI

Create a Red Hat OpenShift Service on AWS cluster or update an existing one using the ROSA command-line interface (CLI) (rosa) that is configured for AWS Spot instances.

10.1. Prerequisites
Copy link

Before configuring Spot instances for Red Hat OpenShift Service on AWS clusters, you must meet the following prerequisites:

  • You have available AWS service quotas.
  • You have enabled Red Hat OpenShift Service on AWS in the AWS console.
  • You have installed and configured the latest rosa CLI on your installation host. Run rosa version to see your currently installed version of the rosa CLI. If a newer version is available, the CLI provides a link to download this upgrade.
  • You have logged into your Red Hat account by using the rosa CLI.
  • You have created an OIDC configuration.
  • You have verified that the AWS Elastic Load Balancing (ELB) service role exists in your AWS account.
  • You have installed and configured the AWS CLI with appropriate credentials.

AWS Spot instances allow you to use spare EC2 capacity at significantly reduced costs compared to on-demand instances. Red Hat OpenShift Service on AWS supports Spot instances for worker node machine pools, enabling cost optimization for fault-tolerant and stateless workloads.

Red Hat OpenShift Service on AWS provides two operational modes for Spot instance termination handling:

ModeDescriptionAWS infrastructure requiredGraceful drain

Simple

Spot instances work with no additional AWS setup. When AWS reclaims a Spot instance, the node is terminated abruptly. A MachineHealthCheck detects the missing node and provisions a replacement.

None

No

Enhanced

Requires a customer-configured SQS queue and EventBridge rules. The AWS Node Termination Handler receives an approximately 2m warning before interruption and gracefully drains pods, respecting PodDisruptionBudgets.

SQS queue, EventBridge rules, SQS resource policy

Yes

Enhanced Spot termination handling requires three AWS components:

  • A Simple Queue Service (SQS) queue to receive interruption events.
  • EventBridge rules to route EC2 Spot events to the queue.
  • An SQS queue resource policy that grants access to EventBridge and the cluster’s NodePoolManagement role.
Note

Use enhanced mode for any workload that benefits from graceful shutdown before Spot interruptions.

You can set up the required AWS infrastructure using the automated rosa CLI helper. This helper command creates all the required AWS resources in a single step.

These resources include:

  • The SQS queue
  • The red-hat=true tag
  • The EventBridge rules
  • The SQS queue resource policy
Note

This process differs from the manual process when setting up the EventBridge rules: the automated rosa CLI helper combines the necessary interruption warning and rebalance recommendation into a single rule using CloudFormation. Setting up the AWS infrastructure manually requires defining two separate rules: one for interruption handling, and one for the rebalance recommendation.

Procedure

  • Run the following command to create the necessary AWS resources: 

    $ rosa create spot-termination-queue --name <cluster-name> --nodepool-management-role-arn <arn> --mode auto
    Note

    This command produces a queue URL that is neccessary for cluster creation, rosa create cluster --spot-termination-queue-url, or updating your cluster, rosa edit cluster --spot-termination-url, so that your SQS queue and cluster are associated.

You can set up the required AWS infrastructure manually by creating the AWS resources individually.

Note

This process differs from the automated process when setting up the EventBridge rules: the automated ROSA command-line interface (CLI) (rosa) helper combines the necessary interruption warning and rebalance recommendation into a single rule using CloudFormation. Setting up the AWS infrastructure manually requires defining two separate rules: one for interruption handling, and one for the rebalance recommendation.

Procedure

  1. Create the SQS queue by running the following command: 

    $ QUEUE_URL=$(aws sqs create-queue \
  --queue-name "$<cluster-name>-spot-interruption-queue" \
  --region "us-east-1" \
  --output text --query 'QueueUrl')

  2. Tag the SQS queue with the red-hat=true tag: 

    $ aws sqs tag-queue-resource \
  --queue-url "${QUEUE_URL}" \
  --tags Key=red-hat,Value=true
    Important

    You must include this tag. Omitting this tag will cause the AWS Node Termination Handler to fail when polling the queue.

  3. Retrieve the queue ARN: 

    $ ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) && \
  QUEUE_ARN="arn:aws:sqs:us-east-1:${ACCOUNT_ID}:<cluster-name>-spot-interruption-queue"

  4. Create two separate EventBridge rules:

    1. Create one rule for Spot interruption warnings: 

      $ aws events put-rule \
  --name "<cluster-name>-spot-interruption" \
  --event-pattern '{"source": ["aws.ec2"], "detail-type": ["EC2 Spot Instance Interruption Warning"]}' \
  --region "us-east-1"

    2. Create another rule for rebalance recommendations: 

      $ aws events put-rule \
  --name "<cluster-name>-rebalance-recommendation" \
  --event-pattern '{"source": ["aws.ec2"], "detail-type": ["EC2 Instance Rebalance Recommendation"]}' \
  --region "us-east-1"

  5. Set the EventBridge target so both rules are routed to the SQS queue: 

    $ aws events put-targets \
  --rule "<cluster-name>-spot-interruption" \
  --targets "Id=1,Arn=${QUEUE_ARN}" \
  --region "us-east-1"
     
    $ aws events put-targets \
  --rule "<cluster-name>-rebalance-recommendation" \
  --targets "Id=1,Arn=${QUEUE_ARN}" \
  --region "us-east-1"

  6. Apply the SQS queue resource policy:

    This policy grants access to two principals:

    EventBridge
    to send interruption events to the queue
    NodePoolManagement role

    to receive and delete messages from the queue

    1. Determine the NodePoolManagement role ARN. The ARN typically follows this pattern: 

      arn:aws:iam::<account-id>:role/ManagedOpenShift-HCP-ROSA-NodePoolManagement-Role

    2. Optional: Retrieve the ARN from an existing cluster: 

      $ aws iam list-roles --query 'Roles[?contains(RoleName, `kube-system-capa-controller-manager`)].Arn' --output text

    3. Set the role ARN and apply the policy: 

      $ NODEPOOL_ROLE_ARN="arn:aws:iam::${ACCOUNT_ID}:role/ManagedOpenShift-HCP-ROSA-NodePoolManagement-Role"
       
      $ cat > /tmp/sqs-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowNodePoolManagementRole",
      "Effect": "Allow",
      "Principal": {
        "AWS": "${NODEPOOL_ROLE_ARN}"
      },
      "Action": [
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage"
      ],
      "Resource": "${QUEUE_ARN}"
    },
    {
      "Sid": "AllowEventBridgeToSendMessages",
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "${QUEUE_ARN}"
    }
  ]
}
EOF
       
      $ aws sqs set-queue-attributes \
  --queue-url "${QUEUE_URL}" \
  --attributes "Policy=$(cat /tmp/sqs-policy.json)"
      Note

      This resource policy provides defense-in-depth alongside the managed IAM policy. The IAM policy restricts access by the tag red-hat=true. The resource policy restricts access by principal ARN.

After you have set up your AWS resources including the SQS queue, EventBridge rules, and associated policy, you can create a new cluster. The process for simple mode and enhanced mode differ by the omission or inclusion respectively of the --spot-termination-queue-url flag.

Important

Each Red Hat OpenShift Service on AWS cluster must use a dedicated Spot termination queue. Do not reuse the same SQS queue URL for multiple clusters. Create a separate queue for each cluster using the rosa create spot-termination-queue command.

Procedure

  • To create a cluster with Spot instances in simple mode, no additional AWS resources are required. Create the cluster without the --spot-termination-queue-url flag: 

    $ rosa create cluster --cluster-name=<cluster-name> \
  --mode=auto --hosted-cp [--private] \
  --operator-roles-prefix <operator-role-prefix> \
  --external-id <external-id> \
  --oidc-config-id <id-of-oidc-configuration> \
  --subnet-ids=<public-subnet-id>,<private-subnet-id>
    Warning

    In simple mode, Spot instances are not gracefully drained upon interruption. Nodes are terminated abruptly when AWS reclaims capacity. A MachineHealthCheck provisions replacement nodes reactively.

  • To create a cluster with Spot instances in enhance mode, create the cluster with a reference to the SQS queue: 

    $ rosa create cluster --cluster-name=<cluster-name> \
  --mode=auto --hosted-cp [--private] \
  --operator-roles-prefix <operator-role-prefix> \
  --external-id <external-id> \
  --oidc-config-id <id-of-oidc-configuration> \
  --subnet-ids=<public-subnet-id>,<private-subnet-id> \
  --spot-termination-queue-url "${QUEUE_URL}"

    The AWS Node Termination Handler is automatically created in the hosted control plane namespace once the cluster is ready and at least one Spot NodePool exists.

10.6. Create a Spot machine pool
Copy link

You can create Spot machine pools for new clusters, or existing ones. This process is identical for both simple and enhanced modes.

Procedure

  • Create a Spot machine pool with fixed replicas: 

    $ rosa create machinepool --cluster=<cluster-name> \
  --name spot-workers \
  --instance-type m5.xlarge \
  --replicas 3 \
  --use-spot-instances \
  --spot-max-price 0.50

  • Alternatively, create a Spot machine pool with autoscaling: 

    $ rosa create machinepool --cluster=<cluster-name> \
  --name spot-workers \
  --instance-type m5.xlarge \
  --enable-autoscaling \
  --min-replicas 2 \
  --max-replicas 10 \
  --use-spot-instances \
  --spot-max-price 0.75
    Note

    Omitting the --spot-max-price flag defaults to the on-demand price.

You can upgrade an existing simple-mode cluster to enhanced mode by adding the AWS resources and associate the SQS queue URL with the cluster.

Prerequisites

  • You have SQS queue configured.
  • You have your EventBridge rules configured.
  • You have the associated resource policy configured

Procedure

  • Associate the queue URL with your existing cluster: 

    $ rosa edit cluster --cluster=<cluster-name> \
  --spot-termination-queue-url "${QUEUE_URL}"

    Existing Spot machine pools immediately benefit from graceful termination handling. The AWS Node Termination Handler deployment is created in the control plane namespace and begins polling the queue. No machine pool changes are required.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top