Chapter 3. Creating a Red Hat OpenShift Service on AWS cluster with Terraform

Procedure

1. Optional: Because the Terraform files get created in your current directory during this procedure, you can create a new directory to store these files and navigate into it by running the following command:

   $ mkdir terraform-cluster && cd terraform-cluster

2. Grant permissions to your account by using an offline Red Hat OpenShift Cluster Manager token.

3. Copy your offline token, and set the token as an environmental variable by running the following command:

   $ export RHCS_TOKEN=<your_offline_token>

   Note

   This environmental variable resets at the end of each session, such as restarting your machine or closing the terminal.

Procedure

1. Create the main.tf file by running the following command:

   $ cat<<-EOF>main.tf
   #
   # Copyright (c) 2023 Red Hat, Inc.
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #   http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   #
   terraform {
     required_providers {
       aws = {
         source  = "hashicorp/aws"
         version = ">= 4.21.0"
       }
       rhcs = {
         version = ">= 1.6.3"
         source  = "terraform-redhat/rhcs"
       }
     }
   }
   
   # Export token using the RHCS_TOKEN environment variable
   provider "rhcs" {}
   
   provider "aws" {
     region = var.aws_region
     ignore_tags {
       key_prefixes = ["kubernetes.io/"]
     }
     default_tags {
       tags = var.default_aws_tags
     }
   }
   
   data "aws_availability_zones" "available" {}
   
   locals {
     # Extract availability zone names for the specified region, limit it to 3 if multi az or 1 if single
     region_azs = var.multi_az ? slice([for zone in data.aws_availability_zones.available.names : format("%s", zone)], 0, 3) : slice([for zone in data.aws_availability_zones.available.names : format("%s", zone)], 0, 1)
   }
   
   resource "random_string" "random_name" {
     length  = 6
     special = false
     upper   = false
   }
   
   locals {
     worker_node_replicas = var.multi_az ? 3 : 2
     # If cluster_name is not null, use that, otherwise generate a random cluster name
     cluster_name = coalesce(var.cluster_name, "rosa-\${random_string.random_name.result}")
   }
   
   # The network validator requires an additional 60 seconds to validate Terraform clusters.
   resource "time_sleep" "wait_60_seconds" {
     count = var.create_vpc ? 1 : 0
     depends_on = [module.vpc]
     create_duration = "60s"
   }
   
   module "rosa-hcp" {
     source                 = "terraform-redhat/rosa-hcp/rhcs"
     version                = "1.6.3"
     cluster_name           = local.cluster_name
     openshift_version      = var.openshift_version
     account_role_prefix    = local.cluster_name
     operator_role_prefix   = local.cluster_name
     replicas               = local.worker_node_replicas
     aws_availability_zones = local.region_azs
     create_oidc            = true
     private                = var.private_cluster
     aws_subnet_ids         = var.create_vpc ? var.private_cluster ? module.vpc[0].private_subnets : concat(module.vpc[0].public_subnets, module.vpc[0].private_subnets) : var.aws_subnet_ids
     create_account_roles   = true
     create_operator_roles  = true
   # Optional: Configure a cluster administrator user
   #
   # Option 1: Default cluster-admin user
   # Create an administrator user (cluster-admin) and automatically
   # generate a password by uncommenting the following parameter:
   #  create_admin_user = true
   # Generated administrator credentials are displayed in terminal output.
   #
   # Option 2: Specify administrator username and password
   # Create an administrator user and define your own password
   # by uncommenting and editing the values of the following parameters:
   #  admin_credentials_username = <username>
   #  admin_credentials_password = <password>
   
   # Optional: Enable cluster deletion protection to prevent accidental deletion.
   # To enable, uncomment the following parameter:
   #  delete_protection = true
   # By default, clusters are created with deletion protection disabled.
   
     depends_on = [time_sleep.wait_60_seconds]
   }
   EOF

   • If you want to create an administrator user during cluster creation, uncomment the appropriate parameters in the Optional: Configure a cluster administrator user section and edit their values.

   • If you want to enable deletion protection during cluster creation, uncomment the corresponding parameter in the Optional: Enable cluster delete protection section. When enabled, deletion protection blocks the terraform destroy command until you set delete_protection = false and apply the change by running terraform apply.

     Deletion protection only blocks deletion requests made through Terraform. It does not set any AWS-level resource protections, and you can still delete cluster resources directly in AWS, risking data loss or cluster outage. Additionally, deletion protection does not override the Red Hat Deletion policy. Red Hat reserves the right to delete clusters that violate service requirements.

2. Create the variables.tf file by running the following command:

   Note

   Copy and edit this file before running the command to build your cluster.

   $ cat<<-EOF>variables.tf
   #
   # Copyright (c) 2023 Red Hat, Inc.
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #   http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   #
   variable "openshift_version" {
     type        = string
     default     = "4.14.20"
     description = "Desired version of OpenShift for the cluster, for example '4.14.20'. If version is greater than the currently running version, an upgrade will be scheduled."
   }
   
   variable "create_vpc" {
     type        = bool
     description = "If you would like to create a new VPC, set this value to 'true'. If you do not want to create a new VPC, set this value to 'false'."
   }
   
   # Red Hat OpenShift Service on AWS cluster info
   variable "cluster_name" {
     default     = null
     type        = string
     description = "The name of the Red Hat OpenShift Service on AWS cluster to create"
   }
   
   variable "additional_tags" {
     default = {
       Terraform   = "true"
       Environment = "dev"
     }
     description = "Additional AWS resource tags"
     type        = map(string)
   }
   
   variable "multi_az" {
     type        = bool
     description = "Multi AZ Cluster for High Availability"
     default     = true
   }
   
   variable "worker_node_replicas" {
     default     = 3
     description = "Number of worker nodes to provision. Single zone clusters need at least 2 nodes, multizone clusters need at least 3 nodes"
     type        = number
   }
   
   variable "aws_subnet_ids" {
     type        = list(any)
     description = "A list of either the public or public + private subnet IDs to use for the cluster blocks to use for the cluster"
     default     = ["subnet-01234567890abcdef", "subnet-01234567890abcdef", "subnet-01234567890abcdef"]
   }
   
   variable "private_cluster" {
     type        = bool
     description = "If you want to create a private cluster, set this value to 'true'. If you want a publicly available cluster, set this value to 'false'."
   }
   
   #VPC Info
   variable "vpc_name" {
     type        = string
     description = "VPC Name"
     default     = "tf-qs-vpc"
   }
   
   variable "vpc_cidr_block" {
     type        = string
     description = "value of the CIDR block to use for the VPC"
     default     = "10.0.0.0/16"
   }
   
   variable "private_subnet_cidrs" {
     type        = list(any)
     description = "The CIDR blocks to use for the private subnets"
     default     = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
   }
   
   variable "public_subnet_cidrs" {
     type        = list(any)
     description = "The CIDR blocks to use for the public subnets"
     default     = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]
   }
   
   variable "single_nat_gateway" {
     type        = bool
     description = "Single NAT or per NAT for subnet"
     default     = false
   }
   
   #AWS Info
   variable "aws_region" {
     type    = string
     default = "us-east-2"
   }
   
   variable "default_aws_tags" {
     type        = map(string)
     description = "Default tags for AWS"
     default     = {}
   }
   EOF

3. Create the vpc.tf file by running the following command:

   $ cat<<-EOF>vpc.tf
   #
   # Copyright (c) 2023 Red Hat, Inc.
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #   http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   #
   module "vpc" {
     source  = "terraform-aws-modules/vpc/aws"
     version = "5.1.2"
   
     count = var.create_vpc ? 1 : 0
     name  = var.vpc_name
     cidr  = var.vpc_cidr_block
   
     azs             = local.region_azs
     private_subnets = var.multi_az ? var.private_subnet_cidrs : [var.private_subnet_cidrs[0]]
     public_subnets  = var.multi_az ? var.public_subnet_cidrs : [var.public_subnet_cidrs[0]]
   
     enable_nat_gateway   = true
     single_nat_gateway   = var.single_nat_gateway
     enable_dns_hostnames = true
     enable_dns_support   = true
   
     tags = var.additional_tags
   }
   EOF

   You are ready to initiate Terraform.

Procedure

1. Configure Terraform to create your resources based on your Terraform files, run the following command:

   $ terraform init

2. Optional: Verify that the Terraform you copied is correct by running the following command:

   $ terraform validate

   Example output

   Success! The configuration is valid.

3. Create your cluster with Terraform by running the following command:

   $ terraform apply

   The Terraform interface asks two questions to create your cluster, similar to the following:

   var.create_vpc
     If you would like to create a new VPC, set this value to 'true'. If you do not want to create a new VPC, set this value to 'false'.
   
     Enter a value:
   
   var.private_cluster
     If you want to create a private cluster, set this value to 'true'. If you want a publicly available cluster, set this value to 'false'.
   
     Enter a value:

4. Enter yes to proceed or no to cancel when the Terraform interface lists the resources to be created or changed and prompts for confirmation:

   Plan: 63 to add, 0 to change, 0 to destroy.
   
   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.

   If you enter yes, your Terraform plan starts, creating your AWS account roles, Operator roles, and your Red Hat OpenShift Service on AWS cluster.

Verification

1. Verify that your cluster was created by running the following command:

   $ rosa list clusters

   This example shows a cluster in the ready state:

   ID                                NAME          STATE  TOPOLOGY
   27c3snjsupa9obua74ba8se5kcj11269  rosa-tf-demo  ready  Hosted CP

2. Verify that your account roles were created by running the following command:

   $ rosa list account-roles

   This example shows the account roles that were created:

   I: Fetching account roles
   ROLE NAME                                   ROLE TYPE      ROLE ARN                                                           OPENSHIFT VERSION  AWS Managed
   ROSA-demo-Installer-Role                    Installer      arn:aws:iam::<ID>:role/ROSA-demo-Installer-Role                    4.14               No
   ROSA-demo-Support-Role                      Support        arn:aws:iam::<ID>:role/ROSA-demo-Support-Role                      4.14               No
   ROSA-demo-Worker-Role                       Worker         arn:aws:iam::<ID>:role/ROSA-demo-Worker-Role                       4.14               No

3. Verify that your Operator roles were created by running the following command:

   $ rosa list operator-roles

   This example shows the Terraform-created Operator roles:

   I: Fetching operator roles
   ROLE PREFIX    AMOUNT IN BUNDLE
   rosa-demo      8

Procedure

1. Create the htpasswd_idp.tf file by running one of the following commands:

   • Option 1: To create a user with a generated, randomized password, run:

     $ cat<<-EOF>htpasswd_idp.tf
       module "htpasswd_idp" {
         source = "terraform-redhat/rosa-hcp/rhcs//modules/idp"
         version = "1.6.2"
     
         cluster_id         = "2odpb9p344hnkfvpkluo00qmgkika78l"
         name               = "htpasswd-idp-tf-1"
         idp_type           = "htpasswd"
         htpasswd_idp_users = [{ username = "pej-user-d1", password = random_password.password.result }]
       }
     
       resource "aws_secretsmanager_secret" "idp_password" {
       name        = "idp-password-secret"
       description = "Any description here"
       }
     
       resource "random_password" "password" {
           length           = 16
           lower            = true
           special          = true
           override_special = "!#$%&*()-_=+[]{}<>:?"
       }
     
       # If you need to output the password, mark it as sensitive to hide from CLI logs
       output "password_output" {
           value     = random_password.password.result
           sensitive = true
       }
     
       # This section sends your credentials to your AWS Secrets Manager to enable you to log in to your cluster.
       resource "aws_secretsmanager_secret_version" "idp_password_val" {
       secret_id     = aws_secretsmanager_secret.idp_password.id
       secret_string = random_password.password.result
       }
     EOF

     You must replace the <cluster_id> placeholder with the 32-digit ID for your cluster. To find that value, run rosa list clusters | awk '{print $1}'. You also must replace the <user_name> placeholder with the username you want to create. The randomized password is then stored in your AWS Secrets manager to be used when logging in to the cluster.

     • Run the following command to view your password after setting it:

       $ terraform output password_output

       The CLI returns your generated password in plain text.

   • Option 2: To specify your passwords when creating a user, run:

     $ cat<<-EOF>htpasswd_idp.tf
       module "htpasswd_idp" {
         source = "terraform-redhat/rosa-hcp/rhcs//modules/idp"
         version = "1.6.2"
     
         cluster_id         = "<cluster_id>"
         name               = "htpasswd-idp"
         idp_type           = "htpasswd"
         htpasswd_idp_users = [{ username="<user_name>",password="<password>"}]
       }
     EOF

     You must replace the <cluster_id> placeholder with the 32-digit ID for your cluster. To find that value, run rosa list clusters | awk '{print $1}'. You also must replace the <user_name> placeholder with the username you want to create and a password for the <password> placeholder.

2. Run the following command to configure Terraform to create your resources based on your Terraform files:

   $ terraform init

3. Verify that the Terraform you copied is correct by running the following command:

   $ terraform validate

   Example output

   Success! The configuration is valid.

4. Create your cluster with Terraform by running the following command:

   $ terraform apply

5. Enter yes to proceed or no to cancel when the Terraform interface lists the resources to be created or changed and prompts for confirmation:

   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.
   
     Enter a value: yes

   You see a confirmation that your IDP has been created.

   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

   Note

   If you used the randomized password template, then the generated password is stored in your AWS Secrets manager.

Procedure

1. In the directory where you ran the terraform apply command to create your cluster, run the following command to delete the cluster:

   $ terraform destroy

   The Terraform interface prompts you for two variables. These should match the answers you provided when creating a cluster:

   var.create_vpc
     If you would like to create a new VPC, set this value to 'true.' If you do not want to create a new VPC, set this value to 'false.'
   
     Enter a value:
   
   var.private_cluster
     If you want to create a private cluster, set this value to 'true.' If you want a publicly available cluster, set this value to 'false.'
   
     Enter a value:

2. Enter yes to start the role and cluster deletion:

   Example output

   Plan: 0 to add, 0 to change, 63 to destroy.
   
   Do you really want to destroy all resources?
     Terraform will destroy all your managed infrastructure, as shown above.
     There is no undo. Only 'yes' will be accepted to confirm.
   
     Enter a value: yes

Verification

1. Verify that your cluster was destroyed by running the following command:

   $ rosa list clusters

   Example output showing no cluster

   I: No clusters available

2. Verify that the account roles were destroyed by running the following command:

   $ rosa list account-roles

   Example output showing no Terraform-created account roles

   I: Fetching account roles
   I: No account roles available

3. Verify that the Operator roles were destroyed by running the following command:

   $ rosa list operator-roles

   Example output showing no Terraform-created Operator roles

   I: Fetching operator roles
   I: No operator roles available

Procedure

1. (Optional): Retrieve the NodePoolManagement role ARN from an existing cluster:

   $ aws iam list-roles --query 'Roles[?contains(RoleName, `kube-system-capa-controller-manager`)].Arn' --output text

2. Add the following variables to your variables.tf file:

   variable "cluster_name" {
     description = "Name of the ROSA HCP cluster"
     type        = string
   }
   
   variable "nodepool_management_role_arn" {
     description = "ARN of the NodePoolManagement IAM role for the cluster"
     type        = string
   }

   The NodePoolManagement role ARN typically follows this pattern:

   arn:aws:iam::<account-id>:role/ManagedOpenShift-HCP-ROSA-NodePoolManagement-Role