×

After installing OpenShift Container Platform on Amazon Web Services (AWS), you can further configure AWS Local Zones and an edge compute pool, so that you can expand and customize your cluster to meet your needs.

Extend existing clusters to use AWS Local Zones

As a postinstallation task, you can extend an existing OpenShift Container Platform cluster on Amazon Web Services (AWS) to use AWS Local Zones.

Extending nodes to Local Zone locations comprises the following steps:

  • Adjusting the cluster-network maximum transmission unit (MTU)

  • Opting in the Local Zone group to AWS Local Zones

  • Creating a subnet in the existing VPC for a Local Zone location

  • Creating the machine set manifest, and then creating a node in each Local Zone location

Before you extend an existing OpenShift Container Platform cluster on AWS to use Local Zones, check that the existing VPC contains available Classless Inter-Domain Routing (CIDR) blocks. These blocks are needed for creating the subnets.

Additional resources
  • For more information about AWS Local Zones, the supported instances types, and services, see AWS Local Zones features in the AWS documentation.

Edge compute pools and AWS Local Zones

Edge worker nodes are tainted worker nodes that run in AWS Local Zones locations.

When deploying a cluster that uses Local Zones, consider the following points:

  • Amazon EC2 instances in the Local Zones are more expensive than Amazon EC2 instances in the Availability Zones.

  • Latency between applications and end users is lower in Local Zones, and latency might vary by location. A latency impact exists for some workloads if, for example, ingress traffic is mixed between Local Zones and Availability Zones.

Generally, the maximum transmission unit (MTU) between an Amazon EC2 instance in a Local Zone and an Amazon EC2 instance in the Region is 1300. For more information, see How Local Zones work in the AWS documentation. The cluster network MTU must be always less than the EC2 MTU to account for the overhead. The specific overhead is determined by the network plugin, for example:

  • OVN-Kubernetes: 100 bytes

  • OpenShift SDN: 50 bytes

The network plugin can provide additional features, like IPsec, that also must be decreased the MTU. For additional information, see the documentation.

OpenShift Container Platform 4.12 introduced a new compute pool, edge, that is designed for use in remote zones. The edge compute pool configuration is common between AWS Local Zones locations. Because of the type and size limitations of resources like EC2 and EBS on Local Zone resources, the default instance type can vary from the traditional worker pool.

The default Elastic Block Store (EBS) for Local Zone locations is gp2, which differs from the regular worker pool. The instance type used for each Local Zone on edge compute pool also might differ from worker pools, depending on the instance offerings on the zone.

The edge compute pool creates new labels that developers can use to deploy applications onto AWS Local Zones nodes. The new labels are:

  • node-role.kubernetes.io/edge=''

  • machine.openshift.io/zone-type=local-zone

  • machine.openshift.io/zone-group=$ZONE_GROUP_NAME

By default, the machine sets for the edge compute pool defines the taint of NoSchedule to prevent regular workloads from spreading on Local Zone instances. Users can only run user workloads if they define tolerations in the pod specification.

Changing the Cluster Network MTU to support AWS Local Zones subnets

You might need to change the maximum transmission unit (MTU) value for the cluster network so that your cluster infrastructure can support Local Zone subnets.

About the cluster MTU

During installation the maximum transmission unit (MTU) for the cluster network is detected automatically based on the MTU of the primary network interface of nodes in the cluster. You do not usually need to override the detected MTU.

You might want to change the MTU of the cluster network for several reasons:

  • The MTU detected during cluster installation is not correct for your infrastructure.

  • Your cluster infrastructure now requires a different MTU, such as from the addition of nodes that need a different MTU for optimal performance.

You can change the cluster MTU for only the OVN-Kubernetes and OpenShift SDN cluster network plugins.

Service interruption considerations

When you initiate an MTU change on your cluster the following effects might impact service availability:

  • At least two rolling reboots are required to complete the migration to a new MTU. During this time, some nodes are not available as they restart.

  • Specific applications deployed to the cluster with shorter timeout intervals than the absolute TCP timeout interval might experience disruption during the MTU change.

MTU value selection

When planning your MTU migration there are two related but distinct MTU values to consider.

  • Hardware MTU: This MTU value is set based on the specifics of your network infrastructure.

  • Cluster network MTU: This MTU value is always less than your hardware MTU to account for the cluster network overlay overhead. The specific overhead is determined by your network plugin:

    • OVN-Kubernetes: 100 bytes

    • OpenShift SDN: 50 bytes

If your cluster requires different MTU values for different nodes, you must subtract the overhead value for your network plugin from the lowest MTU value that is used by any node in your cluster. For example, if some nodes in your cluster have an MTU of 9001, and some have an MTU of 1500, you must set this value to 1400.

How the migration process works

The following table summarizes the migration process by segmenting between the user-initiated steps in the process and the actions that the migration performs in response.

Table 1. Live migration of the cluster MTU
User-initiated steps OpenShift Container Platform activity

Set the following values in the Cluster Network Operator configuration:

  • spec.migration.mtu.machine.to

  • spec.migration.mtu.network.from

  • spec.migration.mtu.network.to

Cluster Network Operator (CNO): Confirms that each field is set to a valid value.

  • The mtu.machine.to must be set to either the new hardware MTU or to the current hardware MTU if the MTU for the hardware is not changing. This value is transient and is used as part of the migration process. Separately, if you specify a hardware MTU that is different from your existing hardware MTU value, you must manually configure the MTU to persist by other means, such as with a machine config, DHCP setting, or a Linux kernel command line.

  • The mtu.network.from field must equal the network.status.clusterNetworkMTU field, which is the current MTU of the cluster network.

  • The mtu.network.to field must be set to the target cluster network MTU and must be lower than the hardware MTU to allow for the overlay overhead of the network plugin. For OVN-Kubernetes, the overhead is 100 bytes and for OpenShift SDN the overhead is 50 bytes.

If the values provided are valid, the CNO writes out a new temporary configuration with the MTU for the cluster network set to the value of the mtu.network.to field.

Machine Config Operator (MCO): Performs a rolling reboot of each node in the cluster.

Reconfigure the MTU of the primary network interface for the nodes on the cluster. You can use a variety of methods to accomplish this, including:

  • Deploying a new NetworkManager connection profile with the MTU change

  • Changing the MTU through a DHCP server setting

  • Changing the MTU through boot parameters

N/A

Set the mtu value in the CNO configuration for the network plugin and set spec.migration to null.

Machine Config Operator (MCO): Performs a rolling reboot of each node in the cluster with the new MTU configuration.

Changing the cluster MTU

As a cluster administrator, you can change the maximum transmission unit (MTU) for your cluster. The migration is disruptive and nodes in your cluster might be temporarily unavailable as the MTU update rolls out.

Prerequisites
  • You installed the OpenShift CLI (oc).

  • You are logged in to the cluster with a user with cluster-admin privileges.

  • You identified the target MTU for your cluster. The correct MTU varies depending on the network plugin that your cluster uses:

    • OVN-Kubernetes: The cluster MTU must be set to 100 less than the lowest hardware MTU value in your cluster.

    • OpenShift SDN: The cluster MTU must be set to 50 less than the lowest hardware MTU value in your cluster.

Procedure

To increase or decrease the MTU for the cluster network complete the following procedure.

  1. To obtain the current MTU for the cluster network, enter the following command:

    $ oc describe network.config cluster
    Example output
    ...
    Status:
      Cluster Network:
        Cidr:               10.217.0.0/22
        Host Prefix:        23
      Cluster Network MTU:  1400
      Network Type:         OpenShiftSDN
      Service Network:
        10.217.4.0/23
    ...
  2. To begin the MTU migration, specify the migration configuration by entering the following command. The Machine Config Operator performs a rolling reboot of the nodes in the cluster in preparation for the MTU change.

    $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
      '{"spec": { "migration": { "mtu": { "network": { "from": <overlay_from>, "to": <overlay_to> } , "machine": { "to" : <machine_to> } } } } }'

    where:

    <overlay_from>

    Specifies the current cluster network MTU value.

    <overlay_to>

    Specifies the target MTU for the cluster network. This value is set relative to the value for <machine_to> and for OVN-Kubernetes must be 100 less and for OpenShift SDN must be 50 less.

    <machine_to>

    Specifies the MTU for the primary network interface on the underlying host network.

    Example that increases the cluster MTU
    $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
      '{"spec": { "migration": { "mtu": { "network": { "from": 1400, "to": 9000 } , "machine": { "to" : 9100} } } } }'
  3. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

    $ oc get mcp

    A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

    By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

  4. Confirm the status of the new machine configuration on the hosts:

    1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"
      Example output
      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.

      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

    2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/mtu-migration.sh
  5. To finalize the MTU migration, enter one of the following commands:

    • If you are using the OVN-Kubernetes network plugin:

      $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
        '{"spec": { "migration": null, "defaultNetwork":{ "ovnKubernetesConfig": { "mtu": <mtu> }}}}'

      where:

      <mtu>

      Specifies the new cluster network MTU that you specified with <overlay_to>.

    • If you are using the OpenShift SDN network plugin:

      $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
        '{"spec": { "migration": null, "defaultNetwork":{ "openshiftSDNConfig": { "mtu": <mtu> }}}}'

      where:

      <mtu>

      Specifies the new cluster network MTU that you specified with <overlay_to>.

Verification
  • Verify that the node in your cluster uses the MTU that you specified in the previous procedure by entering the following command:

    $ oc describe network.config cluster

Opting in to AWS Local Zones

If you plan to create the subnets in AWS Local Zones, you must opt in to each zone group separately.

Prerequisites
  • You have installed the AWS CLI.

  • You have determined an AWS Region for where you want to deploy your OpenShift Container Platform cluster.

  • You have attached a permissive IAM policy to a user or role account that opts in to the zone group. Consider the following configuration as an example IAM policy:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Action": [
            "ec2:ModifyAvailabilityZoneGroup"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
      ]
    }
Procedure
  1. List the zones that are available in your AWS Region by running the following command:

    $ aws --region "<value_of_AWS_Region>" ec2 describe-availability-zones \
        --query 'AvailabilityZones[].[{ZoneName: ZoneName, GroupName: GroupName, Status: OptInStatus}]' \
        --filters Name=zone-type,Values=local-zone \
        --all-availability-zones

    Depending on the AWS Region, the list of available zones can be long. The command returns the following fields:

    ZoneName

    The name of the Local Zone.

    GroupName

    The group that comprises the zone. To opt in to the region, save the name.

    Status

    The status of the Local Zone group. If the status is not-opted-in, you must opt in the GroupName by running the commands that follow.

  2. Opt in to the zone group on your AWS account by running the following command:

    $ aws ec2 modify-availability-zone-group \
        --group-name "<value_of_GroupName>" \(1)
        --opt-in-status opted-in
    1 For <value_of_GroupName>, specify the name of the group of the Local Zone where you want to create subnets. For example, specify us-east-1-nyc-1 to use the zone us-east-1-nyc-1a (US East New York).

Extend existing clusters to use AWS Local Zones

If you want a Machine API to create an Amazon EC2 instance in a remote zone location, you must create a subnet in a Local Zone location. You can use any provisioning tool, such as Ansible or Terraform, to create subnets in the existing Virtual Private Cloud (VPC). You can configure the CloudFormation template to meet your requirements.

The following subsections include steps that use CloudFormation templates. Considering the limitation of NAT Gateways in AWS Local Zones, CloudFormation templates support only public subnets. You can reuse the template to create public subnets for each edge location to where you need to extend your cluster.

Creating a subnet in AWS Local Zones

You must create a subnet in AWS Local Zones before you configure a worker machineset for your OpenShift Container Platform cluster.

You must repeat the following process for each Local Zone you want to deploy worker nodes to.

You can use the provided CloudFormation template and a custom parameter file to create a stack of AWS resources that represent the subnet.

If you do not use the provided CloudFormation template to create your AWS infrastructure, you must review the provided information and manually create the infrastructure. If your cluster does not initialize correctly, you might have to contact Red Hat support with your installation logs.

Prerequisites
  • You configured an AWS account.

  • You added your AWS keys and region to your local AWS profile by running aws configure.

  • You opted in to the Local Zone group.

Procedure
  1. Create a JSON file that contains the parameter values that the template requires:

    [
      {
        "ParameterKey": "VpcId",
        "ParameterValue": "<value_of_VpcId>" (1)
      },
      {
        "ParameterKey": "PublicRouteTableId",
        "ParameterValue": "<value_of_PublicRouteTableId>" (2)
      },
      {
        "ParameterKey": "ZoneName",
        "ParameterValue": "<value_of_ZoneName>" (3)
      },
      {
        "ParameterKey": "SubnetName",
        "ParameterValue": "<value_of_SubnetName>"
      },
      {
        "ParameterKey": "PublicSubnetCidr",
        "ParameterValue": "10.0.192.0/20" (4)
      }
    ]
    1 Specify the VPC ID, which is the value VpcID in the output of the CloudFormation template. for the VPC.
    2 Specify the Route Table ID, which is the value of the PublicRouteTableId in the CloudFormation stack for the VPC.
    3 Specify the AWS Local Zone name, which is the value of the ZoneName field in the AvailabilityZones object that you retrieve in the section "Opting in to AWS Local Zones".
    4 Specify a CIDR block that is used to create the Local Zone subnet. This block must be part of the VPC CIDR block VpcCidr.
  2. Copy the template from the CloudFormation template for the subnet section of this topic and save it as a YAML file on your computer. This template describes the VPC that your cluster requires.

  3. Launch the CloudFormation template to create a stack of AWS resources that represent the VPC by running the following command:

    You must enter the command on a single line.

    $ aws cloudformation create-stack --stack-name <subnet_stack_name> \ (1)
         --template-body file://<template>.yaml \ (2)
         --parameters file://<parameters>.json (3)
    1 <subnet_stack_name> is the name for the CloudFormation stack, such as cluster-lz-<local_zone_shortname>. You need the name of this stack if you remove the cluster.
    2 <template> is the relative path to and name of the CloudFormation template YAML file that you saved.
    3 <parameters> is the relative path to and name of the CloudFormation parameters JSON file.
    Example output
    arn:aws:cloudformation:us-east-1:123456789012:stack/<subnet_stack_name>/dbedae40-2fd3-11eb-820e-12a48460849f
  4. Confirm that the template components exist by running the following command:

    $ aws cloudformation describe-stacks --stack-name <subnet_stack_name>

    After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. You must provide these parameter values to the other CloudFormation templates that you run to create your cluster:

    PublicSubnetIds

    The IDs of the new public subnets.

CloudFormation template for the subnet that uses AWS Local Zones

You can use the following CloudFormation template to deploy the subnet that you need for your OpenShift Container Platform cluster that uses AWS Local Zones.

CloudFormation template for the subnet
# CloudFormation template used to create Local Zone subnets and dependencies
AWSTemplateFormatVersion: 2010-09-09
Description: Template for create Public Local Zone subnets

Parameters:
  VpcId:
    Description: VPC Id
    Type: String
  ZoneName:
    Description: Local Zone Name (Example us-east-1-nyc-1a)
    Type: String
  SubnetName:
    Description: Local Zone Name (Example cluster-public-us-east-1-nyc-1a)
    Type: String
  PublicRouteTableId:
    Description: Public Route Table ID to associate the Local Zone subnet
    Type: String
  PublicSubnetCidr:
    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(1[6-9]|2[0-4]))$
    ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/16-24.
    Default: 10.0.128.0/20
    Description: CIDR block for Public Subnet
    Type: String

Resources:
  PublicSubnet:
    Type: "AWS::EC2::Subnet"
    Properties:
      VpcId: !Ref VpcId
      CidrBlock: !Ref PublicSubnetCidr
      AvailabilityZone: !Ref ZoneName
      Tags:
      - Key: Name
        Value: !Ref SubnetName
      - Key: kubernetes.io/cluster/unmanaged
        Value: "true"

  PublicSubnetRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      SubnetId: !Ref PublicSubnet
      RouteTableId: !Ref PublicRouteTableId

Outputs:
  PublicSubnetIds:
    Description: Subnet IDs of the public subnets.
    Value:
      !Join ["", [!Ref PublicSubnet]]

Creating a machine set manifest for an AWS Local Zones node

After you create subnets in AWS Local Zones, you can create a machine set manifest.

The installation program sets the following labels for the edge machine pools at cluster installation time:

  • machine.openshift.io/parent-zone-name: <value_of_ParentZoneName>

  • machine.openshift.io/zone-group: <value_of_ZoneGroup>

  • machine.openshift.io/zone-type: <value_of_ZoneType>

The following procedure details how you can create a machine set configuraton that matches the edge compute pool configuration.

Prerequisites
  • You have created subnets in AWS Local Zones.

Procedure
  • Manually preserve edge machine pool labels when creating the machine set manifest by gathering the AWS API. To complete this action, enter the following command in your command-line interface (CLI):

    $ aws ec2 describe-availability-zones --region <value_of_Region> \(1)
        --query 'AvailabilityZones[].{
    	ZoneName: ZoneName,
    	ParentZoneName: ParentZoneName,
    	GroupName: GroupName,
    	ZoneType: ZoneType}' \
        --filters Name=zone-name,Values=<value_of_ZoneName> \(2)
        --all-availability-zones
    1 For <value_of_Region>, specify the name of the region for the zone.
    2 For <value_of_ZoneName>, specify the name of the Local Zone.
Example output for Local Zone us-east-1-nyc-1a
[
    {
        "ZoneName": "us-east-1-nyc-1a",
        "ParentZoneName": "us-east-1f",
        "GroupName": "us-east-1-nyc-1",
        "ZoneType": "local-zone"
    }
]

Sample YAML for a compute machine set custom resource on AWS

This sample YAML defines a compute machine set that runs in the us-east-1-nyc-1a Amazon Web Services (AWS) zone and creates nodes that are labeled with node-role.kubernetes.io/edge: "".

In this sample, <infrastructure_id> is the infrastructure ID label that is based on the cluster ID that you set when you provisioned the cluster, and <edge> is the node label to add.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
  name: <infrastructure_id>-edge-<zone> (2)
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-edge-<zone>
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        machine.openshift.io/cluster-api-machine-role: edge (3)
        machine.openshift.io/cluster-api-machine-type: edge (3)
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-edge-<zone> (2)
    spec:
      metadata:
        labels:
          machine.openshift.io/parent-zone-name: <value_of_ParentZoneName>
          machine.openshift.io/zone-group: <value_of_GroupName>
          machine.openshift.io/zone-type: <value_of_ZoneType>
          node-role.kubernetes.io/edge: "" (3)
      providerSpec:
        value:
          ami:
            id: ami-046fe691f52a953f9 (4)
          apiVersion: machine.openshift.io/v1beta1
          blockDevices:
            - ebs:
                iops: 0
                volumeSize: 120
                volumeType: gp2
          credentialsSecret:
            name: aws-cloud-credentials
          deviceIndex: 0
          iamInstanceProfile:
            id: <infrastructure_id>-worker-profile (1)
          instanceType: m6i.large
          kind: AWSMachineProviderConfig
          placement:
            availabilityZone: <zone> (6)
            region: <region> (7)
          securityGroups:
            - filters:
                - name: tag:Name
                  values:
                    - <infrastructure_id>-worker-sg (1)
          subnet:
              id: <value_of_PublicSubnetIds> (8)
          publicIp: true
          tags:
            - name: kubernetes.io/cluster/<infrastructure_id> (1)
              value: owned
            - name: <custom_tag_name> (5)
              value: <custom_tag_value> (5)
          userDataSecret:
            name: worker-user-data
      taints: (9)
        - key: node-role.kubernetes.io/edge
          effect: NoSchedule
1 Specify the infrastructure ID that is based on the cluster ID that you set when you provisioned the cluster. If you have the OpenShift CLI installed, you can obtain the infrastructure ID by running the following command:
$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
2 Specify the infrastructure ID, edge role node label, and zone name.
3 Specify the edge role node label.
4 Specify a valid Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI) for your AWS zone for your OpenShift Container Platform nodes. If you want to use an AWS Marketplace image, you must complete the OpenShift Container Platform subscription from the AWS Marketplace to obtain an AMI ID for your region.
$ oc -n openshift-machine-api \
    -o jsonpath='{.spec.template.spec.providerSpec.value.ami.id}{"\n"}' \
    get machineset/<infrastructure_id>-<role>-<zone>
5 Optional: Specify custom tag data for your cluster. For example, you might add an admin contact email address by specifying a name:value pair of Email:admin-email@example.com.

Custom tags can also be specified during installation in the install-config.yml file. If the install-config.yml file and the machine set include a tag with the same name data, the value for the tag from the machine set takes priority over the value for the tag in the install-config.yml file.

6 Specify the zone name, for example, us-east-1-nyc-1a.
7 Specify the region, for example, us-east-1.
8 The ID of the public subnet that you created in AWS Local Zones. You created this public subnet ID on completing the procedure for "Creating a subnet in AWS Local Zones".
9 Specify a taint to prevent user workloads from being scheduled on edge nodes.

After adding the NoSchedule taint on the infrastructure node, existing DNS pods running on that node are marked as misscheduled. You must either delete or add toleration on misscheduled DNS pods.

Creating a compute machine set

In addition to the compute machine sets created by the installation program, you can create your own to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites
  • Deploy an OpenShift Container Platform cluster.

  • Install the OpenShift CLI (oc).

  • Log in to oc as a user with cluster-admin permission.

Procedure
  1. Create a new YAML file that contains the compute machine set custom resource (CR) sample and is named <file_name>.yaml.

    Ensure that you set the <clusterID> and <role> parameter values.

  2. Optional: If you are not sure which value to set for a specific field, you can check an existing compute machine set from your cluster.

    1. To list the compute machine sets in your cluster, run the following command:

      $ oc get machinesets -n openshift-machine-api
      Example output
      NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
      agl030519-vplxk-worker-us-east-1a   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1b   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1c   1         1         1       1           55m
      agl030519-vplxk-worker-us-east-1d   0         0                             55m
      agl030519-vplxk-worker-us-east-1e   0         0                             55m
      agl030519-vplxk-worker-us-east-1f   0         0                             55m
    2. To view values of a specific compute machine set custom resource (CR), run the following command:

      $ oc get machineset <machineset_name> \
        -n openshift-machine-api -o yaml
      Example output
      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: <infrastructure_id> (1)
        name: <infrastructure_id>-<role> (2)
        namespace: openshift-machine-api
      spec:
        replicas: 1
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: <infrastructure_id>
            machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: <infrastructure_id>
              machine.openshift.io/cluster-api-machine-role: <role>
              machine.openshift.io/cluster-api-machine-type: <role>
              machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
          spec:
            providerSpec: (3)
              ...
      1 The cluster infrastructure ID.
      2 A default node label.

      For clusters that have user-provisioned infrastructure, a compute machine set can only create worker and infra type machines.

      3 The values in the <providerSpec> section of the compute machine set CR are platform-specific. For more information about <providerSpec> parameters in the CR, see the sample compute machine set CR configuration for your provider.
  3. Create a MachineSet CR by running the following command:

    $ oc create -f <file_name>.yaml
Verification
  • View the list of compute machine sets by running the following command:

    $ oc get machineset -n openshift-machine-api
    Example output
    NAME                                       DESIRED   CURRENT   READY   AVAILABLE   AGE
    agl030519-vplxk-edge-us-east-1-nyc-1a      1         1         1       1           11m
    agl030519-vplxk-worker-us-east-1a          1         1         1       1           55m
    agl030519-vplxk-worker-us-east-1b          1         1         1       1           55m
    agl030519-vplxk-worker-us-east-1c          1         1         1       1           55m
    agl030519-vplxk-worker-us-east-1d          0         0                             55m
    agl030519-vplxk-worker-us-east-1e          0         0                             55m
    agl030519-vplxk-worker-us-east-1f          0         0                             55m

    When the new compute machine set is available, the DESIRED and CURRENT values match. If the compute machine set is not available, wait a few minutes and run the command again.

  • Optional: To check nodes that were created by the edge machine, run the following command:

    $ oc get nodes -l node-role.kubernetes.io/edge
    Example output
    NAME                           STATUS   ROLES         AGE    VERSION
    ip-10-0-207-188.ec2.internal   Ready    edge,worker   172m   v1.25.2+d2e245f

Creating user workloads in AWS Local Zones

After you create an Amazon Web Service (AWS) Local Zone environment, and you deploy your cluster, you can use edge worker nodes to create user workloads in Local Zone subnets.

After you run the installation program and create the cluster, the installation program automatically specifies a taint effect of NoSchedule to each edge worker node. This means that a scheduler does not add a new pod, or deployment, to a node if the pod does not match the specified tolerations for a taint. You can modify the taint for better control over how nodes create workloads in each Local Zone subnet.

The installation program creates the compute machine set manifests file with node-role.kubernetes.io/edge and node-role.kubernetes.io/worker labels applied to each edge worker node that is located in a Local Zone subnet.

Prerequisites
  • You have access to the OpenShift CLI (oc).

  • You deployed your cluster in a Virtual Private Cloud (VPC) with defined Local Zone subnets.

  • You ensured that the compute machine set for the edge workers on Local Zone subnets specifies the taints for node-role.kubernetes.io/edge.

Procedure
  1. Create a deployment resource YAML file for an example application to be deployed in the edge worker node that operates in a Local Zone subnet. Ensure that you specify the correct tolerations that match the taints for the edge worker node.

    Example of a configured deployment resource for an edge worker node that operates in a Local Zone subnet
    kind: Namespace
    apiVersion: v1
    metadata:
      name: <local_zone_application_namespace>
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: <pvc_name>
      namespace: <local_zone_application_namespace>
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: gp2-csi (1)
      volumeMode: Filesystem
    ---
    apiVersion: apps/v1
    kind: Deployment (2)
    metadata:
      name: <local_zone_application> (3)
      namespace: <local_zone_application_namespace> (4)
    spec:
      selector:
        matchLabels:
          app: <local_zone_application>
      replicas: 1
      template:
        metadata:
          labels:
            app: <local_zone_application>
            zone-group: ${ZONE_GROUP_NAME} (5)
        spec:
          securityContext:
            seccompProfile:
              type: RuntimeDefault
          nodeSelector: (6)
            machine.openshift.io/zone-group: ${ZONE_GROUP_NAME}
          tolerations: (7)
          - key: "node-role.kubernetes.io/edge"
            operator: "Equal"
            value: ""
            effect: "NoSchedule"
          containers:
            - image: openshift/origin-node
              command:
               - "/bin/socat"
              args:
                - TCP4-LISTEN:8080,reuseaddr,fork
                - EXEC:'/bin/bash -c \"printf \\\"HTTP/1.0 200 OK\r\n\r\n\\\"; sed -e \\\"/^\r/q\\\"\"'
              imagePullPolicy: Always
              name: echoserver
              ports:
                - containerPort: 8080
              volumeMounts:
                - mountPath: "/mnt/storage"
                  name: data
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: <pvc_name>
    1 storageClassName: For the Local Zone configuration, you must specify gp2-csi.
    2 kind: Defines the deployment resource.
    3 name: Specifies the name of your Local Zone application. For example, local-zone-demo-app-nyc-1.
    4 namespace: Defines the namespace for the AWS Local Zone where you want to run the user workload. For example: local-zone-app-nyc-1a.
    5 zone-group: Defines the group to where a zone belongs. For example, us-east-1-iah-1.
    6 nodeSelector: Targets edge worker nodes that match the specified labels.
    7 tolerations: Sets the values that match with the taints defined on the MachineSet manifest for the Local Zone node.
  2. Create a service resource YAML file for the node. This resource exposes a pod from a targeted edge worker node to services that run inside your Local Zone network.

    Example of a configured service resource for an edge worker node that operates in a Local Zone subnet
    apiVersion: v1
    kind: Service (1)
    metadata:
      name:  <local_zone_application>
      namespace: <local_zone_application_namespace>
    spec:
      ports:
        - port: 80
          targetPort: 8080
          protocol: TCP
      type: NodePort
      selector: (2)
        app: <local_zone_application>
    1 kind: Defines the service resource.
    2 selector: Specifies the label type applied to managed pods.
Next steps
  • Optional: Use the AWS Load Balancer (ALB) Operator to expose a pod from a targeted edge worker node to services that run inside of a Local Zone subnet from a public network. See Installing the AWS Load Balancer Operator.