Understand storage specific requirements and supported features for distributed compute node (DCN) deployments. These considerations help you plan storage configurations that work within DCN architectural constraints.

The following features are not currently supported for DCN architectures:

• Copying a volume between edge sites. You can work around this by creating an image from the volume and using the Image service (glance) to copy the image. After the image is copied, you can create a volume from it.
• Ceph Rados Gateway (RGW) at the edge sites.
• CephFS at the edge sites.
• Instance high availability (HA) at the edge sites.
• RBD mirroring between edge sites.
• Instance migration, live or cold, either between edge sites, or from the central location to edge sites. You can only migrate instances within a site boundary. To move an image between sites, you must snapshot the image, and use glance image-import.

Additionally, you must consider the following:

• You must upload images to the central location before copying them to edge sites. A copy of each image must exist in the Image service at the central location.
• You must use the RBD storage driver for the Image, Compute and Block Storage services.
• For each site, including the central location, assign a unique availability zone.
• You can migrate an offline volume from an edge site to the central location, or vice versa. You cannot migrate volumes directly between edge sites.

Understand architectural limitations and requirements for DCN architecture deployments. These considerations help you successfully deploy distributed edge locations and maintain performance.

The following features are not currently supported for DCN architectures:

• DHCP on DPDK nodes
• TC Flower Hardware Offload

The following ML2/OVN networking technologies are fully supported:

• Routed provider networks
• OVN GW (network node) with Neutron AZs supported

Additionally, you must consider the following:

• Network latency: Balance the latency as measured in round-trip time (RTT), with the expected number of concurrent API operations to maintain acceptable performance. Maximum TCP/IP throughput is inversely proportional to RTT. You can mitigate some issues with high-latency connections with high bandwidth by tuning kernel TCP parameters. Contact Red Hat Support if a cross-site communication exceeds 100 ms.
• Network drop outs: If the edge site temporarily loses connection to the central site, then no control plane API or CLI operations can be executed at the impacted edge site for the duration of the outage. For example, Compute nodes at the edge site are consequently unable to create a snapshot of an instance, issue an auth token, or delete an image. General control plane API and CLI operations remain functional during this outage, and can continue to serve any other edge sites that have a working connection.
• Image type: You must use raw images when deploying a DCN architecture with Ceph storage.
• Image sizing:
  • Compute images: Compute images are downloaded from the central location. These images are potentially large files that are transferred across all necessary networks from the central site to the edge site during provisioning.
  • Instance images: If there is no block storage at the edge, then the Image service images traverse the WAN during first use. The images are copied or cached locally to the target edge nodes for all subsequent use. There is no size limit for images. Transfer times vary with available bandwidth and network latency.
• Provider networks are the most common approach for DCN deployments. Note that the Networking service (neutron) does not validate where you can attach available networks. For example, if you use a provider network called "site-a" only in edge site A, the Networking service does not validate and prevent you from attaching "site-a" to an instance at site B, which does not work.
• Site-specific networks: A limitation in DCN networking arises if you use networks that are specific to a certain site: When you deploy centralized neutron controllers with Compute nodes, there are no triggers in the Networking service to identify a certain Compute node as a remote node. Consequently, the Compute nodes receive a list of other Compute nodes and automatically form tunnels between each other. The tunnels are formed from edge to edge through the central site. If you use GENEVE, every Compute node at every site forms a tunnel with every other Compute node, whether or not they are local or remote. This is not an issue if you are using the same networks everywhere. When you use VLANs, the Networking service expects that all Compute nodes have the same bridge mappings, and that all VLANs are available at every site.
• If edge servers are not pre-provisioned, you must configure DHCP relay for introspection and provisioning on routed segments.
• Routing must be configured either on the cloud or within the networking infrastructure that connects each edge site to the hub. You should implement a networking design that allocates an L3 subnet for each RHOSO cluster network (external, internal API, and so on), unique to each site. If you are using BGP, you must configure BGP on the routers in these locations to learn the routes advertised by the RHOSO control plane and data plane nodes.

Size your internal API network address pool based on the number of distributed compute node (DCN) sites in your deployment. Each site requires dedicated IP addresses for Image service (glance) endpoints and load balancers.

The Image service operator creates an endpoint for each Image service pod with its own DNS name, such as glance-az0-internal.openstack.svc:9292. Each Compute service and Block storage service in each availability zone uses the Image service API server in the same availability zone. For example, when you update the cinderVolumes field in the OpenStackControlPlane custom resource (CR), add a field called glance_api_servers under customServiceConfig:

      cinderVolumes:
        az0:
          customServiceConfig: |
            [DEFAULT]
            enabled_backends = az0
            glance_api_servers = https://glance-az0-internal.openstack.svc:9292

The Image service endpoint DNS name maps to a load-balancer IP address in the internalapi address pool as indicated by the internal metadata annotations:

            [glance_store]
            default_backend = ceph
            [ceph]
            rbd_store_ceph_conf = /etc/ceph/ceph.conf
            store_description = "ceph RBD backend"
            rbd_store_pool = images
            rbd_store_user = openstack
            rbd_thin_provisioning = True
          networkAttachments:
          - storage
          override:
            service:
              internal:
                metadata:
                  annotations:
                    metallb.universe.tf/address-pool: internalapi
                    metallb.universe.tf/allow-shared-ip: internalapi
                    metallb.universe.tf/loadBalancerIPs: 172.17.0.80

The range of addresses in this address pool should be sized according to the number of DCN sites. For example, the following shows only 10 available addresses in the internalapi network.

$ oc get ipaddresspool -n metallb-system
NAME          AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
ctlplane      true          false             ["192.168.122.80-192.168.122.90"]
internalapi   true          false             ["172.17.0.80-172.17.0.90"]
storage       true          false             ["172.18.0.80-172.18.0.90"]
tenant        true          false             ["172.19.0.80-172.19.0.90"]

Use commands such as the following after updating the glance section of the OpenStackControlPlane CR to confirm that the Glance Operator has created the service endpoint and route.

$ oc get svc | grep glance
glance-az0-internal             LoadBalancer   172.30.217.178   172.17.0.80      9292:32134/TCP                                   24h
glance-az0-public               ClusterIP      172.30.78.47     <none>           9292/TCP                                         24h
glance-az1-internal             LoadBalancer   172.30.52.123    172.17.0.81      9292:31679/TCP                                   23h
glance-c1ca8-az0-external-api   ClusterIP      None             <none>           9292/TCP                                         24h
glance-c1ca8-az0-internal-api   ClusterIP      None             <none>           9292/TCP                                         24h
glance-c1ca8-az1-edge-api       ClusterIP      None             <none>           9292/TCP                                         23h

$ oc get route | grep glance
glance-az0-public              glance-az0-public-openstack.apps.ocp.openstack.lab                     glance-az0-public              glance-az0-public              reencrypt/Redirect   None
glance-default-public          glance-default-public-openstack.apps.ocp.openstack.lab                 glance-default-public          glance-default-public          reencrypt/Redirect   None