Chapter 1. BGP routing

Procedure

1. Get the OpenID Connect (OIDC) provider and AWS account ID for the cluster by entering the following commands:

   $ OIDC_PROVIDER=$(oc get authentication cluster -o jsonpath='{.spec.serviceAccountIssuer}' | sed 's|https://||')

   $ AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

2. Create an AWS IAM permission policy for the BGP Cloud Connector Operator.

   1. Create the permission policy document file by entering the following command:

      $ cat <<EOF > operator-permission-policy.json
      {
        "Version": "2012-10-17",
        "Statement": [{
          "Effect": "Allow",
          "Action": [
            "sts:GetCallerIdentity",
            "ec2:DescribeRouteServers",
            "ec2:DescribeRouteServerEndpoints",
            "ec2:DescribeSubnets",
            "ec2:DescribeRouteServerPeers",
            "ec2:CreateRouteServerPeer",
            "ec2:DeleteRouteServerPeer",
            "ec2:CreateTags",
            "ec2:DescribeInstances",
            "ec2:ModifyNetworkInterfaceAttribute"
          ],
          "Resource": "*"
        }]
      }
      EOF

   2. Create the IAM permission policy by entering the following command:

      $ POLICY_ARN=$(aws iam create-policy \
          --policy-name bgp-cloud-connector-operator-policy \
          --policy-document file://operator-permission-policy.json \
          --query 'Policy.Arn' --output text)

3. Create an AWS IAM role for the BGP Cloud Connector Operator:

   1. Create the trust policy document for the Operator role by entering the following command:

      $ cat <<EOF > operator-trust-policy.json
      {
        "Version": "2012-10-17",
        "Statement": [{
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "${OIDC_PROVIDER}:sub": "system:serviceaccount:bgp-cloud-connector:openshift-bgp-cloud-connector-controller-manager"
            }
          }
        }]
      }
      EOF

   2. Create the IAM role by using the trust policy by entering the following command:

      $ aws iam create-role --role-name bgp-cloud-connector-operator \
          --assume-role-policy-document file://operator-trust-policy.json

   3. Attach the permission policy to the IAM role by entering the following command:

      $ aws iam attach-role-policy \
          --role-name bgp-cloud-connector-operator \
          --policy-arn ${POLICY_ARN}

4. Create a dedicated security group for BGP traffic in your cluster’s VPC by entering the following command:

   $ BGP_SG_ID=$(aws ec2 create-security-group \
       --group-name "bgp-peering-sg" \
       --description "Security group for BGP Cloud Connector peering traffic" \
       --vpc-id ${vpc_id} \
       --query 'GroupId' --output text)

   Replace vpc_id with the ID of the VPC of the cluster.

5. Add rules to the security group to allow the following BGP traffic and workload traffic patterns. You can add any ports to the security group that your VMs or pods on the CUDN need for traffic. You can edit this security group at a later stage.

   Note

   If you plan to use Bidirectional Forwarding Detection (BFD), repeat the commands to allow the appropriate UDP control traffic ports,such as UDP port 3784. Restrict the --cidr range to your specific Route Server endpoint subnets if required by the security policies of your organization.

   • Allow TCP Ingress/Egress on port 179 for bidirectional BGP traffic by running the following command:

     $ aws ec2 authorize-security-group-egress --group-id ${BGP_SG_ID} --ip-permissions '[
       {"IpProtocol": "tcp", "FromPort": 179, "ToPort": 179, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "BGP"}]}
     ]'

   • Allow TCP/UDP Ingress traffic on ports 32768-60999, because bidirectional traffic originating from a VM might enter the cluster through a different compute node than the bidirectional traffic exits. Run the following command:

     $ aws ec2 authorize-security-group-ingress --group-id ${BGP_SG_ID} --ip-permissions '[
       {"IpProtocol": "tcp", "FromPort": 179, "ToPort": 179, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "BGP"}]},
       {"IpProtocol": "tcp", "FromPort": 32768, "ToPort": 60999, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Ephemeral"}]},
       {"IpProtocol": "udp", "FromPort": 32768, "ToPort": 60999, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Ephemeral"}]}
     ]'

6. Create one or more new Red Hat OpenShift Service on AWS machine pools that include the BGP security group and the required BGP router label by running the following command:

   $ rosa create machinepool \
       --cluster <cluster_name> \
       --name bgp-workers \
       --availability-zone <availability_zone> \
       --additional-security-group-ids ${BGP_SG_ID} \
       --labels "bgp_router=true" \
       --instance-type <instance_type> \
       --replicas 2

   where:

   <cluster_name>

   Specifies the name of the cluster.

   bgp-workers

   Specifies the name of the new machine pool dedicated to BGP workloads.

   <availability_zone>

   Specifies the AWS availability zone where the machine pool gets created.

   --additional-security-group-ids

   Attaches the BGP security group created in an earlier step to all compute nodes provisioned in the machine pool.

   --labels "bgp_router=true"

   Automatically labels all compute nodes in this machine pool to participate in BGP.

   <instance_type>

   Specifies the AWS instance type of the compute nodes.

   --replicas 2

   Specifies the number of compute nodes the machine pool should have.

7. Verify that the new machine pool was created with the additional security group and label by entering the following command:

   $ rosa describe machinepool --cluster <cluster_name> bgp-workers

   Replace <cluster_name> with the name of your cluster.

8. Confirm that the compute nodes provisioned by the machine pool inherit the bgp_router=true label by running the following command:

   $ oc get nodes -l bgp_router=true

9. Create an AWS Route Server by entering the following command:

   $ aws ec2 create-route-server --amazon-side-asn <asn>

   Replace <asn> with the Border Gateway Protocol (BGP) Autonomous System Number (ASN) that you want the Route Server to use. Note the RouteServerId value in the command output that is required in a later step. In the example that follows, the ID is rs-0abc123456789abcd.

   {
      "RouteServer": {
          "RouteServerId": "rs-0abc123456789abcd",
          "AmazonSideAsn": 65100,
          "State": "pending",
          "PersistRoutes": "disable",
          "SnsNotificationsEnabled": false
      }
   }

10. Associate the Route Server with the VPC of the cluster by entering the following command:

    $ aws ec2 associate-route-server --route-server-id <route_server_id> \
       --vpc-id <vpc_id>

    where:

    <route_server_id>

    Specifies the Route Server ID from an earlier step.

    <vpc_id>

    Specifies the ID of the VPC of the cluster.

11. Find the private subnets used by the compute nodes of the cluster by entering the following command:

    $ rosa list machinepools --cluster <cluster_name>

    Replace <cluster_name> with the name of the cluster.

    Note

    From the output of the command, ignore any duplicate subnet IDs in the SUBNET column

12. For each private subnet that contains BGP-enabled compute nodes, create Route Server endpoints, two per subnet:

    $ aws ec2 create-route-server-endpoint --route-server-id <route_server_id> \
       --subnet-id <subnet_id> && \
       aws ec2 create-route-server-endpoint --route-server-id <route_server_id> \
       --subnet-id <subnet_id>

    Replace <route_server_id> with the ID of your Route Server.

    Replace <subnet_id> with the ID of the subnet. Repeat this command for each private subnet that contains BGP-enabled compute nodes.

    Note

    AWS allows a maximum of two Route Server endpoints per subnet.

13. Enable route propagation from the Route Server to the relevant VPC route tables.

    1. Identify the route tables associated with your VPC by entering the following command:

       $ aws ec2 describe-route-tables \
         --filters "Name=vpc-id,Values=<vpc_id>" \
         --query "RouteTables[].{RouteTableId: RouteTableId, Name: Tags[?Key=='Name'].Value | [0], Subnets: Associations[].SubnetId}" \
         --output table

       Replace <vpc_id> with the ID of the VPC for your cluster.

    2. Enable route propagation for each route table by entering the following command:

       $ aws ec2 enable-route-server-propagation --route-server-id <route_server_id> \
         --route-table-id <route_table_id>

       Replace <route_server_id> with the ID of your Route Server, and replace <route_table_id> with the ID of a route table identified in an earlier step.

       Repeat this command for all of the following route tables:

       • Compute node subnets (Minimum): Every route table associated with the private subnets containing your compute nodes.
       • External connectivity subnets (If applicable): Any route table associated with subnets that host network interfaces for external connectivity mechanisms into the VPC, such as Transit Gateway ENIs.

Procedure

1. To install the BGP Cloud Connector Operator by using the OpenShift CLI (oc), complete the following steps:

   1. Create a dedicated namespace and an OperatorGroup manifest file for the BGP Cloud Connector Operator by entering the following command:

      $ oc create namespace bgp-cloud-connector

   2. Create an OperatorGroup manifest file in the dedicated namespace by entering the following command:

      $ echo '{"apiVersion":"operators.coreos.com/v1","kind":"OperatorGroup","metadata":{"name":"bgp-cloud-connector-operatorgroup","namespace":"bgp-cloud-connector"}, "spec":{"targetNamespaces":["bgp-cloud-connector"]}}' | oc apply -f -

   3. Configure AWS authentication for the Operator based on the credential mode of your cluster. The Cloud Credential Operator (CCO) automatically manages AWS credentials for the Operator based on the credential mode of your cluster.

      • STS Mode: Pass your pre-created IAM role ARN to the Operator by setting the ROLEARN environment variable in the Subscription custom resource (CR) or specifying it in the Role ARN field in the web console.

        If you are deploying the Operator manually through a Subscription CR, set the ROLEARN environment variable in your manifest:

        apiVersion: operators.coreos.com/v1alpha1
        kind: Subscription
        metadata:
          name: bgp-cloud-connector-operator
          namespace: bgp-cloud-connector
        spec:
          channel: stable
          name: bgp-cloud-connector
          source: redhat-operators
          sourceNamespace: openshift-marketplace
          config:
            env:
              - name: ROLEARN
                value: arn:aws:iam::<aws_account_id>:role/<iam_role_name>

        Note

        If you install the Operator by using the web console, the console automatically prompts you for the Role ARN and then populates ROLEARN for you.

        You must create the IAM role in AWS before setting ROLEARN. The Operator cannot generate its own IAM role. Ensure the trust policy of the role trusts the OIDC provider of your cluster.

   4. Verify that the Operator installation succeeded by entering the following command:

      $ oc get csv -n bgp-cloud-connector

      Confirm that the PHASE column displays Succeeded.

2. To install BGP Cloud Connector Operator by using the web console, complete the following steps:

   1. Log in to the web console as a cluster administrator.
   2. Go to Ecosystem Software Catalog
   3. Search for BGP cloud connector.
   4. Click the Install button.
   5. Select bgp-cloud-connector as the namespace.
   6. Click the Install button.

Procedure

1. Create a BGPCloudConfiguration custom resource (CR) to define the cluster-wide BGP configuration:

   apiVersion: networking.openshift.io/v1beta1
   kind: BGPCloudConfiguration
   metadata:
     name: cluster
   spec:
     platform: AWS
     bgp:
       localASN: <local_asn>
       livenessDetection: bgp-keepalive
     routerNodeSelector:
       bgp_router: "true"
     aws:
       region: <aws_region>
       routeServerIDs:
         - <route_server_id>

   where:

   spec.bgp.localASN.<local_asn>

   Specifies the Autonomous System Number (ASN) for the FRRouting (FRR) router of your cluster. The <local_asn> value must be different from the Route Server ASN that you configured with --amazon-side-asn, which the Operator discovers automatically. Ensure you note the ARN of the role from the output.

   spec.routerNodeSelector.bgp_router

   Specifies the label key used in the routerNodeSelector to identify which compute nodes participate in BGP routing. Nodes that match this selector have the following characteristics:

   • Establish BGP peering sessions with Route Server Endpoints in their local subnet.
   • Act as the next-hop for BGP-advertised routes for traffic that ingresses from the VPC to the primary Cluster User Default Network (CUDN) of the workload.

   • Have source/destination checks disabled on their Elastic Network Interfaces (ENIs).

     Important

     Traffic always egresses from pods or VMs directly through their own compute node ENI. Therefore, you must apply this selector to any node where these workloads can be scheduled, not just the nodes intended to receive incoming traffic.

   spec.aws.region.<aws_region>

   Specifies the AWS region in which cloud resources get created. Replace <aws_region> with the AWS Region of your Route Server.

   spec.aws.routeServerIDs.<route_server_id>

   Specifies the Route Server ID. Replace <route_server_id> with the Route Server ID that you noted while preparing the AWS environment.

2. Apply the updates to the BGP cloud configuration resource on your cluster by entering the following command:

   $ oc apply -f bgpcloudconfiguration.yaml

   Note

   BGPCloudConfiguration is a cluster-scoped singleton. You must provide the name cluster to the CR.

3. Create a namespace for your workload. Use the labels that the BGP Cloud Connector requires.

   apiVersion: v1
   kind: Namespace
   metadata:
     name: <namespace_name>
     labels:
       k8s.ovn.org/primary-user-defined-network: ""
       cluster-udn: <network_name>

   Replace <namespace_name> with a name for the namespace. Replace <network_name> with the network name that you plan to use in the following step.

   Important

   You must set the k8s.ovn.org/primary-user-defined-network label when you create the namespace. Red Hat OpenShift Service on AWS admission policy prevents you from adding this label to an existing namespace.

4. Create the Namespace object by running the following command:

   $ oc apply -f namespace.yaml

5. Create a BGPRouting CR to declare the network to advertise over BGP:

   apiVersion: networking.openshift.io/v1beta1
   kind: BGPRouting
   metadata:
     name: <routing_name>
   spec:
     network:
       name: <network_name>
       subnets:
         - <pod_network_cidr>

   where:

   <routing_name>

   Specifies the name for the CR.

   <network_name>

   Specifies the Layer 2 ClusterUserDefinedNetwork to create. The value must be the same value that you used for cluster-udn in an earlier step.

   <pod_network_cidr>

   The CIDR block that the primary network interfaces of each VM get assigned from. For example 10.100.0.0/16. The value is set as the spec.network.layer2.subnets of the CUDN. The value must not overlap with any of the CIDRs in your VPC.

   Note

   BGPRouting resources create a Layer 2 ClusterUserDefinedNetwork based upon the name and subnets you specify. BGP Cloud Connector advertises routes for the subnets to your AWS Route Server.

   The spec.namespaceSelector.matchLabels label for ClusterUserDefinedNetwork is set to cluster-udn: <network_name> to match the namespace you created in an earlier step. The name of the CUDN is cluster-udn-<network_name>.

   Layer 2 CUDNs only allow a single IPv4 and a single IPv6 CIDR per subnet.

6. Apply the BGPRouting CR by running the following command:

   $ oc apply -f bgprouting.yaml

Verification

1. Confirm that both custom resources report a Ready phase by entering the following commands:

   $ oc get bgpcloudconfiguration cluster -o jsonpath='{.status.phase}'

   $ oc get bgprouting <routing_name> -o jsonpath='{.status.phase}'

2. Optional: If either resource reports Degraded, inspect its conditions for the failure reason by running the following command:

   $ oc get bgpcloudconfiguration cluster -o jsonpath='{.status.conditions}' | jq .

3. Confirm that the Operator generated the supporting resources by entering the following commands:

   $ oc get frrconfiguration -n openshift-frr-k8s

   $ oc get clusteruserdefinednetwork

   $ oc get routeadvertisements

4. Deploy a virtual machine into the namespace that you created, and verify that it uses bridge binding on its network interface:

   spec:
     domain:
       devices:
         interfaces:
           - name: default
             bridge: {}
     networks:
       - name: default
         pod: {}

   Important

   Use bridge binding, not masquerade binding. masquerade binding installs nftables rules that drop inbound traffic on the user-defined network (UDN) interface, which blocks the direct connectivity that this feature provides.

5. From an EC2 instance within the VPC, verify that you can reach the VM at its pod network IP address by entering the following command:

   $ ping <vm_pod_ip>

   Replace <vm_pod_ip> with the pod network IP address of the VM.

6. Optional: To verify connectivity from an external network, such as an on-premises network connected over AWS Direct Connect or a VPN, first ensure that the external network can route traffic for the VM pod network into the VPC.