Adopt the data plane

Adopt your existing environment to the Red Hat OpenStack Services on OpenShift (RHOSO) data plane.

Adopting the data plane involves the following steps:

  1. Stop any remaining services on the Red Hat OpenStack Platform (RHOSP) 17.1 control plane.
  2. Deploy the required custom resources.
  3. Perform a fast-forward upgrade on Compute services from RHOSP 17.1 to RHOSO 18.0.
  4. Adopt Networker services to the RHOSO data plane.
  5. Optional: Migrate OVN gateway services from RHOSP 17.1 nodes to Red Hat OpenShift Container Platform (RHOCP) worker nodes to reclaim hardware.
Warning
After the RHOSO control plane manages the newly deployed data plane, you must not re-enable services on the RHOSP 17.1 control plane and data plane. If you re-enable services, workloads are managed by two control planes or two data planes, resulting in data corruption, loss of control of existing workloads, inability to start new workloads, or other issues.

Stop infrastructure management and Compute services
Copy link

You must stop cloud database nodes and messaging nodes on the Red Hat OpenStack Platform 17.1 control plane.

Before you begin
Copy link

  • Define the shell variables. Replace the following example values with values that apply to your environment: 
    CONTROLLER1_SSH="ssh -i <path_to_SSH_key> tripleo-admin@<controller-1 IP>"
# ...
# ...
EDPM_PRIVATEKEY_PATH="<path_to_SSH_key>"
    • CONTROLLER<X>_SSH defines the SSH connection details for all Controller nodes, including cell Controller nodes, of the source director cloud.
    • <path_to_SSH_key> defines the path to your SSH key.

About this task
Copy link

Do not stop nodes that are running the following roles:

  • Compute
  • Storage
  • Networker
  • Controller if running OVN Controller Gateway agent network agent

The following procedure applies to a standalone director deployment. You must stop the Pacemaker services on your host so that you can install libvirt packages when the Compute roles are adopted as data plane nodes. Modular libvirt daemons no longer run in Podman containers on data plane nodes.

Procedure
Copy link

Stop the Pacemaker services: 
PacemakerResourcesToStop=(
                "galera-bundle"
                "haproxy-bundle"
                "rabbitmq-bundle")

echo "Stopping pacemaker services"
for i in {1..3}; do
    SSH_CMD=CONTROLLER${i}_SSH
    if [ ! -z "${!SSH_CMD}" ]; then
        echo "Using controller $i to run pacemaker commands"
        for resource in ${PacemakerResourcesToStop[*]}; do
            if ${!SSH_CMD} sudo pcs resource config $resource; then
                ${!SSH_CMD} sudo pcs resource disable $resource
            fi
        done
        break
    fi
done

Adopt Compute services to the RHOSO data plane
Copy link

Adopt your Compute (nova) services to the Red Hat OpenStack Services on OpenShift (RHOSO) data plane.

Before you begin
Copy link

  • You have stopped the remaining control plane nodes, repositories, and packages on the Compute service (nova) hosts. For more information, see Stopping infrastructure management and Compute services.
  • If you have a Red Hat Ceph Storage environment, you have configured the Ceph back end for the NovaLibvirt service. For more information, see Configuring a Ceph back end.
  • You have configured IP Address Management (IPAM): 
    $ oc apply -f - <<EOF
apiVersion: network.openstack.org/v1beta1
kind: NetConfig
metadata:
  name: netconfig
spec:
  networks:
  - name: ctlplane
    dnsDomain: ctlplane.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 192.168.122.120
        start: 192.168.122.100
      - end: 192.168.122.200
        start: 192.168.122.150
      cidr: 192.168.122.0/24
      gateway: 192.168.122.1
  - name: internalapi
    dnsDomain: internalapi.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.17.0.250
        start: 172.17.0.100
      cidr: 172.17.0.0/24
      vlan: 20
  - name: External
    dnsDomain: external.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 10.0.0.250
        start: 10.0.0.100
      cidr: 10.0.0.0/24
      gateway: 10.0.0.1
  - name: storage
    dnsDomain: storage.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.18.0.250
        start: 172.18.0.100
      cidr: 172.18.0.0/24
      vlan: 21
  - name: storagemgmt
    dnsDomain: storagemgmt.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.20.0.250
        start: 172.20.0.100
      cidr: 172.20.0.0/24
      vlan: 23
  - name: tenant
    dnsDomain: tenant.example.com
    subnets:
    - name: subnet1
      allocationRanges:
      - end: 172.19.0.250
        start: 172.19.0.100
      cidr: 172.19.0.0/24
      vlan: 22
EOF
  • If neutron-sriov-nic-agent is running on your Compute service nodes, ensure that the physical device mappings match the values that are defined in the OpenStackDataPlaneNodeSet custom resource (CR).
  • To prevent workload shutdown, you have created the tripleo_nova_libvirt_guests_service_cleanup.yaml playbook: 
    - become: true
  hosts: all
  strategy: tripleo_free
  name: disable and clean tripleo_nova_libvirt_guests
  tasks:
    - name: tripleo_nova_libvirt_guests removal
      become: true
      shell: |
        set -o pipefail
        systemctl disable tripleo_nova_libvirt_guests.service
        rm -f /etc/systemd/system/tripleo_nova_libvirt_guests.service
        rm -f /etc/systemd/system/virt-guest-shutdown.target
        systemctl daemon-reload

    You have used the following command to run the playbook:

    $ansible-playbook -i overcloud-deploy/overcloud/tripleo-ansible-inventory.yaml tripleo_nova_libvirt_guests_service_cleanup.yaml
  • You have defined the shell variables to run the script that runs the upgrade: 
    $ CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //')

$ alias openstack="oc exec -t openstackclient -- openstack"

$ DEFAULT_CELL_NAME="cell3"
$ RENAMED_CELLS="cell1 cell2 $DEFAULT_CELL_NAME"

$ declare -A COMPUTES_CELL1
$ export COMPUTES_CELL1=(
> ["standalone.localdomain"]="192.168.122.100"
> # <compute1>
> # <compute2>
> # <compute3>
>)
$ declare -A COMPUTES_CELL2
$ export COMPUTES_CELL2=(
> # <compute1>
>)
$ declare -A COMPUTES_CELL3
$export COMPUTES_CELL3=(
> # <compute1>
> # <compute2>
>)

$ declare -A COMPUTES_API_CELL1
$export COMPUTES_API_CELL1=(
> ["standalone.localdomain"]="172.17.0.100"
> ["standalone2.localdomain"]="172.17.0.101"
>)

$ declare -A COMPUTES_STORAGE_CELL1
$ export COMPUTES_STORAGE_CELL1=(
  ["standalone.localdomain"]="172.18.0.100"
  # <compute1-storage-ip>
  # <compute2-storage-ip>
)
$ declare -A COMPUTES_STORAGE_CELL2
$ export COMPUTES_STORAGE_CELL2=(
  # <compute1-storage-ip>
)
$ declare -A COMPUTES_STORAGE_CELL3
$ export COMPUTES_STORAGE_CELL3=(
  # <compute1-storage-ip>
  # <compute2-storage-ip>
)

$ declare -A COMPUTES_TENANT_CELL1
$ export COMPUTES_TENANT_CELL1=(
  ["standalone.localdomain"]="172.19.0.100"
  # <compute1-tenant-ip>
  # <compute2-tenant-ip>
)
$ declare -A COMPUTES_TENANT_CELL2
$ export COMPUTES_TENANT_CELL2=(
  # <compute1-tenant-ip>
)
$ declare -A COMPUTES_TENANT_CELL3
$ export COMPUTES_TENANT_CELL3=(
  # <compute1-tenant-ip>
  # <compute2-tenant-ip>
)

$ NODESETS=""
$ for CELL in $(echo $RENAMED_CELLS); do
> ref="COMPUTES_$(echo ${CELL}|tr '[:lower:]' '[:upper:]')"
> eval names=\${!${ref}[@]}
> [ -z "$names" ] && continue
> NODESETS="'openstack-${CELL}', $NODESETS"
>done
$ NODESETS="[${NODESETS%,*}]"
    • DEFAULT_CELL_NAME="cell3" defines the source cloud default cell that acquires a new DEFAULT_CELL_NAME on the destination cloud after adoption. In a multi-cell adoption scenario, you can retain the original name, default, or create a new cell default name by providing the incremented index of the last cell in the source cloud. For example, if the incremented index of the last cell is cell5, the new cell default name is cell6.
    • export COMPUTES_CELL1= For each cell, update the <["standalone.localdomain"]="x.x.x.x"> value and the COMPUTES_CELL<X> value with the names and IP addresses of the Compute service nodes that are connected to the ctlplane and internalapi networks. Do not specify a real FQDN defined for each network. Always use the same hostname for each connected network of a Compute node. Provide the IP addresses and the names of the hosts on the remaining networks of the source cloud as needed, or you can manually adjust the files that you generate in step 9 of this procedure.
    • <compute1>, <compute2>, and <compute3> specifies the names of your Compute service nodes for each cell. Assign all Compute service nodes from the source cloud cell1 cell into COMPUTES_CELL1, and so on.
    • export COMPUTES_CELL<X>=( specifies all Compute service nodes that you assign from the source cloud default cell into COMPUTES_CELL<X> and COMPUTES_API_CELL<X>, where <X> is the DEFAULT_CELL_NAME environment variable value. In this example, the DEFAULT_CELL_NAME environment variable value equals cell3.
    • export COMPUTES_API_CELL1=( For each cell, update the <["standalone.localdomain"]="192.168.122.100"> value and the COMPUTES_API_CELL<X> value with the names and IP addresses of the Compute service nodes that are connected to the ctlplane and internalapi networks. ["standalone.localdomain"]="192.168.122.100" defines the custom DNS domain in the FQDN value of the nodes. This value is used in the data plane node set spec.nodes.<NODE NAME>.hostName. Do not specify a real FQDN defined for each network. Use the same hostname for each of its connected networks. Provide the IP addresses and the names of the hosts on the remaining networks of the source cloud as needed, or you can manually adjust the files that you generate in step 9 of this procedure.
    • <compute1-storage-ip>, <compute2-storage-ip>: For each cell, replace these values with the names and IP addresses of the Compute service nodes that are connected to the storage network. These IP addresses must match the existing addresses on the 17.1 Compute nodes. Retrieve them with ip addr on each node, from Heat outputs, or from /etc/os-net-config/config.yaml.
    • <compute1-tenant-ip>, <compute2-tenant-ip>: For each cell, replace these values with the names and IP addresses of the Compute service nodes that are connected to the tenant network. These IP addresses must match the existing addresses on the 17.1 Compute nodes. Retrieve them with ip addr on each node, from Heat outputs, or from /etc/os-net-config/config.yaml.
    • NODESETS="'openstack-${CELL}', $NODESETS" specifies the cells that contain Compute nodes. Cells that do not contain Compute nodes are omitted from this template because no node sets are created for the cells. Note

      If you deployed the source cloud with a default cell, and want to rename it during adoption, define the new name that you want to use, as shown in the following example:

      $ DEFAULT_CELL_NAME="cell1"
$ RENAMED_CELLS="cell1"
Note

Do not set a value for the CEPH_FSID parameter if the local storage back end is configured by the Compute service for libvirt. The storage back end must match the source cloud storage back end. You cannot change the storage back end during adoption.

Procedure
Copy link

  1. Create an SSH authentication secret for the data plane nodes: 
    $ oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
    name: dataplane-adoption-secret
data:
    ssh-privatekey: |
$(cat <path_to_SSH_key> | base64 | sed \'s/^/        /')
EOF
    • Replace <path_to_SSH_key> with the path to your SSH key.

      For more information about creating data plane secrets, see Create data plane Secret CRs.

  2. Generate an ssh key-pair nova-migration-ssh-key secret: 
    $ cd "$(mktemp -d)"
$ ssh-keygen -f ./id -t ecdsa-sha2-nistp521 -N ''
$ oc get secret nova-migration-ssh-key || oc create secret generic nova-migration-ssh-key \
  --from-file=ssh-privatekey=id \
  --from-file=ssh-publickey=id.pub \
  --type kubernetes.io/ssh-auth
$ rm -f id*
$ cd -
  3. Create the libvirt secret for Compute nodes:
    Note

    The libvirt-secret is required for all Compute node deployments, regardless of whether TLS Everywhere is enabled. When TLS Everywhere is enabled, use the existing LibvirtTLSPassword from your source deployment. When TLS Everywhere is not enabled, generate a new password.
    1. If TLS Everywhere is enabled, set LIBVIRT_PASSWORD to match the existing RHOSP deployment password: 
      declare -A TRIPLEO_PASSWORDS
TRIPLEO_PASSWORDS[default]="$HOME/overcloud-passwords.yaml"
LIBVIRT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' LibvirtTLSPassword:' | awk -F ': ' '{ print $2; }')
LIBVIRT_PASSWORD_BASE64=$(echo -n "$LIBVIRT_PASSWORD" | base64)
    2. If TLS Everywhere is not enabled, generate a new password: 
      LIBVIRT_PASSWORD=$(openssl rand -base64 12)
LIBVIRT_PASSWORD_BASE64=$(echo -n "$LIBVIRT_PASSWORD" | base64)
    3. Create the libvirt-secret: 
      $ oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: libvirt-secret
type: Opaque
data:
  LibvirtPassword: ${LIBVIRT_PASSWORD_BASE64}
EOF
  4. Create a config map to use for all cells to configure a local storage back end for libvirt: 
    $ oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: nova-cells-global-config
data:
  99-nova-compute-cells-workarounds.conf: |
    [workarounds]
    disable_compute_service_check_for_ffu=true
EOF
    • data provides the configuration files for all the cells.
    • 99-nova-compute-cells-workarounds.conf: | specifies the index of the <*.conf> files. There is a requirement to index the <*.conf> files from '03' to '99', based on precedence. A <99-*.conf> file takes the highest precedence, while indexes below '03' are reserved for internal use. Note
      If you adopt a live cloud, you might be required to carry over additional configurations for the default nova data plane services that are stored in the cell1 default nova-extra-config config map. Do not delete or overwrite the existing configuration in the cell1 default nova-extra-config config map that is assigned to nova. Overwriting the configuration can break the data place services that rely on specific contents of the nova-extra-config config map.
  5. Configure a Red Hat Ceph Storage back end for libvirt: 
    $ oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: nova-cells-global-config
data:
  99-nova-compute-cells-workarounds.conf: |
    [workarounds]
    disable_compute_service_check_for_ffu=true
  03-ceph-nova.conf: |
    [libvirt]
    images_type=rbd
    images_rbd_pool=vms
    images_rbd_ceph_conf=/etc/ceph/ceph.conf
    images_rbd_glance_store_name=default_backend
    images_rbd_glance_copy_poll_interval=15
    images_rbd_glance_copy_timeout=600
    rbd_user=openstack
    rbd_secret_uuid=$CEPH_FSID
EOF
    Note
    For Red Hat Ceph Storage environments with multi-cell configurations, you must name config maps and Red Hat OpenStack Platform data plane services similar to the following examples: nova-custom-ceph-cellX and nova-compute-extraconfig-cellX.
     Note

    For Distributed Compute Node (DCN) deployments, do not use the single nova-cells-global-config ConfigMap. Create a per-site ConfigMap and per-site OpenStackDataPlaneService for each site in your DCN deployment. Each site’s Compute service nodes require a different Red Hat Ceph Storage configuration and a different Image service endpoint. For more information, see Adopting Compute services with multiple Ceph back ends (DCN).
  6. Create the data plane services for Compute service cells to enable pre-upgrade workarounds, and to configure the Compute services for your chosen storage back end: 
    for CELL in $(echo $RENAMED_CELLS); do
oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
 name: nova-$CELL
spec:
 dataSources:
   - secretRef:
       name: nova-$CELL-compute-config
   - secretRef:
       name: nova-migration-ssh-key
   - configMapRef:
       name: nova-cells-global-config
 playbook: osp.edpm.nova
 caCerts: combined-ca-bundle
 edpmServiceType: nova
 containerImageFields:
 - NovaComputeImage
 - EdpmIscsidImage
EOF
 done
    • spec.dataSources.secretRef specifies an additional autogenerated nova-cell<X>-metadata-neutron-config secret to enable a local metadata service for cell<X>. You should also set spec.nova.template.cellTemplates.cell<X>.metadataServiceTemplate.enable in the OpenStackControlPlane/openstack CR, as described in Adopting the Compute service. You can configure a single top-level metadata, or define the metadata per cell.
    • nova-$CELL-compute-config specifies the secret that auto-generates for each cell<X>. You must append the nova-cell<X>-compute-config for each custom OpenStackDataPlaneService CR that is related to the Compute service.
    • nova-migration-ssh-key specifies the secret that you must append for each custom OpenStackDataPlaneService CR that is related to the Compute service. Note

      When creating your data plane services for Compute service cells, review the following considerations:

      • In this example, the same nova-migration-ssh-key key is shared across cells. However, you should use different keys for different cells.
      • For simple configuration overrides, you do not need a custom data plane service. However, to reconfigure the cell, cell1, the safest option is to create a custom service and a dedicated config map for it.
      • The cell, cell1, is already managed with the default OpenStackDataPlaneService CR called nova and its nova-extra-config config map. Do not change the default data plane service nova definition. The changes are lost when the RHOSO operator is updated with OLM.
      • When a cell spans multiple node sets, give the custom OpenStackDataPlaneService resources a name that relates to the node set, for example, nova-cell1-nfv and nova-cell1-enterprise. The autogenerated config maps are then named nova-cell1-nfv-extra-config and nova-cell1-enterprise-extra-config.
      • Different configurations for nodes in multiple node sets of the same cell are also supported, but are not covered in this guide.
  7. If TLS Everywhere is enabled, append the following content to the OpenStackDataPlaneService CR: 
      tlsCerts:
    nova:
      contents:
        - dnsnames
        - ips
      networks:
        - ctlplane
      issuer: osp-rootca-issuer-internal
      edpmRoleServiceName: nova
  caCerts: combined-ca-bundle
  edpmServiceType: nova
  8. Create a secret for the subscription manager: 
    $ oc create secret generic subscription-manager \
--from-literal rhc_auth='{"login": {"username": "<subscription_manager_username>", "password": "<subscription_manager_password>"}}'
    • Replace <subscription_manager_username> with the applicable username.
    • Replace <subscription_manager_password> with the applicable password.
  9. Create a secret for the Red Hat registry: 
    $ oc create secret generic redhat-registry \
--from-literal edpm_container_registry_logins='{"registry.redhat.io": {"<registry_username>": "<registry_password>"}}'
    • Replace <registry_username> with the applicable username.
    • Replace <registry_password> with the applicable password. Note
      You do not need to reference the subscription-manager secret in the dataSources field of the OpenStackDataPlaneService CR. The secret is already passed in with a node-specific OpenStackDataPlaneNodeSet CR in the ansibleVarsFrom property in the nodeTemplate field.
  10. Create the data plane node set definitions for each cell: 
    $ declare -A names
$ for CELL in $(echo $RENAMED_CELLS); do
  ref="COMPUTES_$(echo ${CELL}|tr '[:lower:]' '[:upper:]')"
  eval names=\${!${ref}[@]}
  ref_api="COMPUTES_API_$(echo ${CELL}|tr '[:lower:]' '[:upper:]')"
  ref_storage="COMPUTES_STORAGE_$(echo ${CELL}|tr '[:lower:]' '[:upper:]')"
  ref_tenant="COMPUTES_TENANT_$(echo ${CELL}|tr '[:lower:]' '[:upper:]')"
  [ -z "$names" ] && continue
  ind=0
  rm -f computes-$CELL
  for compute in $names; do
    ip="${ref}['$compute']"
    ip_api="${ref_api}['$compute']"
    ip_storage="${ref_storage}['$compute']"
    ip_tenant="${ref_tenant}['$compute']"
    cat >> computes-$CELL << EOF
    ${compute}:
      hostName: $compute
      ansible:
        ansibleHost: $compute
      networks:
      - defaultRoute: true
        fixedIP: ${!ip}
        name: ctlplane
        subnetName: subnet1
      - name: internalapi
        subnetName: subnet1
        fixedIP: ${!ip_api}
      - name: storage
        subnetName: subnet1
        fixedIP: ${!ip_storage}
      - name: tenant
        subnetName: subnet1
        fixedIP: ${!ip_tenant}
EOF
    ind=$(( ind + 1 ))
  done

  test -f computes-$CELL || continue
  cat > nodeset-${CELL}.yaml <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneNodeSet
metadata:
  name: openstack-$CELL
spec:
  tlsEnabled: false
  networkAttachments:
      - ctlplane
  preProvisioned: true
  services:
    - redhat
    - bootstrap
    - download-cache
    - configure-network
    - validate-network
    - install-os
    - configure-os
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ovn
    - neutron-metadata
    - libvirt
    - nova-$CELL
    - telemetry
  env:
    - name: ANSIBLE_CALLBACKS_ENABLED
      value: "profile_tasks"
    - name: ANSIBLE_FORCE_COLOR
      value: "True"
    - name: ANSIBLE_VERBOSITY
      value: "3"
  nodeTemplate:
    ansibleSSHPrivateKeySecret: dataplane-adoption-secret
    ansible:
      ansibleUser: tripleo-admin
      ansibleVarsFrom:
      - secretRef:
          name: subscription-manager
      - secretRef:
          name: redhat-registry
      ansibleVars:
        rhc_release: 9.2
        rhc_repositories:
            - {name: "*", state: disabled}
            - {name: "<repository_name1>", state: enabled}
            - {name: "<repository_name2>", state: enabled}
            - {name: "<repository_name3>", state: enabled}
            - {name: "<repository_name4>", state: enabled}
            - {name: "<repository_name5>", state: enabled}
            - {name: "<repository_name6>", state: enabled}
        edpm_bootstrap_release_version_package: []
        # edpm_network_config
        # Default nic config template for a EDPM node
        # These vars are edpm_network_config role vars
        edpm_network_config_template: |
           ---
           {% set mtu_list = [ctlplane_mtu] %}
           {% for network in nodeset_networks %}
           {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
           {%- endfor %}
           {% set min_viable_mtu = mtu_list | max %}
           network_config:
           - type: ovs_bridge
             name: {{ neutron_physical_bridge_name }}
             mtu: {{ min_viable_mtu }}
             use_dhcp: false
             dns_servers: {{ ctlplane_dns_nameservers }}
             domain: {{ dns_search_domains }}
             addresses:
             - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
             routes: {{ ctlplane_host_routes }}
             members:
             - type: interface
               name: nic1
               mtu: {{ min_viable_mtu }}
               # force the MAC address of the bridge to this interface
               primary: true
           {% for network in nodeset_networks %}
             - type: vlan
               mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
               vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
               addresses:
               - ip_netmask:
                   {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
           {% endfor %}

        edpm_network_config_nmstate: false
        # Control resolv.conf management by NetworkManager
        # false = disable NetworkManager resolv.conf update (default)
        # true = enable NetworkManager resolv.conf update
        edpm_bootstrap_network_resolvconf_update: false
        edpm_network_config_hide_sensitive_logs: false
        #
        # These vars are for the network config templates themselves and are
        # considered EDPM network defaults.
        neutron_physical_bridge_name: br-ctlplane
        neutron_public_interface_name: eth0

        # edpm_nodes_validation
        edpm_nodes_validation_validate_controllers_icmp: false
        edpm_nodes_validation_validate_gateway_icmp: false

        # edpm ovn-controller configuration
        edpm_ovn_bridge_mappings: [<"bridge_mappings">]
        edpm_ovn_bridge: br-int
        edpm_ovn_encap_type: geneve
        ovn_monitor_all: true
        edpm_ovn_remote_probe_interval: 60000
        edpm_ovn_ofctrl_wait_before_clear: 8000

        timesync_ntp_servers:
        - hostname: clock.redhat.com
        - hostname: clock2.redhat.com
        edpm_bootstrap_command: |
          set -euxo pipefail
          sudo dnf -y upgrade openstack-selinux
          sudo rm -f /run/virtlogd.pid

        gather_facts: false
        # edpm firewall, change the allowed CIDR if needed
        edpm_sshd_configure_firewall: true
        edpm_sshd_allowed_ranges: ['192.168.122.0/24']

        # Do not attempt OVS major upgrades here
        edpm_ovs_packages:
        - openvswitch3.3
        edpm_default_mounts:
          - path: /dev/hugepages<size>
            opts: pagesize=<size>
            fstype: hugetlbfs
            group: hugetlbfs
  nodes:
EOF
  cat computes-$CELL >> nodeset-${CELL}.yaml
done
    • ${compute}.hostName specifies the FQDN for the node if your deployment has a custom DNS Domain.
    • ${compute}.networks specifies the network composition. The network composition must match the source cloud configuration to avoid data plane connectivity downtime. The ctlplane network must come first. The commands only retain IP addresses for the hosts on the ctlplane and internalapi networks. Repeat this step for other isolated networks, or update the resulting files manually.
    • ${compute}.ansible.ansibleHost specifies the Ansible host for the Compute node. If you are not using DNS, use the value ${!ip}.
    • metadata.name: specifies the node set names for each cell, for example, openstack-cell1, openstack-cell2. Only create node sets for cells that contain Compute nodes.
    • spec.tlsEnabled specifies whether TLS Everywhere is enabled. If it is enabled, change tlsEnabled to true.
    • spec.services specifies the services to be adopted. If you are not adopting telemetry services, omit it from the services list.
    • neutron_physical_bridge_name: br-ctlplane specifies the bridge name. The bridge name and other OVN and Networking service-specific values must match the source cloud configuration to avoid data plane connectivity downtime.
    • edpm_ovn_bridge_mappings: Replace [<"bridge_mappings">] with the value of the bridge mappings in your configuration, for example, ["datacentre:br-ctlplane"].
    • path: /dev/hugepages<size> and opts: pagesize=<size> configures huge pages. Replace <size> with the size of the page. To configure multi-sized huge pages, create more items in the list. Note that the mount points must match the source cloud configuration.
    • <repository_name1>, <repository_name2>, <repository_name3>, <repository_name4>, <repository_name5>, <repository_name6> specifies the name of the repository to enable. For more information about which repositories are required, see Required repositories for Red Hat OpenStack Services on OpenShift 18.0. Note

      Ensure that you use the same ovn-controller settings in the OpenStackDataPlaneNodeSet CR that you used in the Compute service nodes before adoption. This configuration is stored in the external_ids column in the Open_vSwitch table in the Open vSwitch database:

      $ ovs-vsctl list Open .
...
external_ids        : {hostname=standalone.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
...
  11. Deploy the OpenStackDataPlaneNodeSet CRs for each Compute cell: 
    for CELL in $(echo $RENAMED_CELLS); do
test -f nodeset-${CELL}.yaml || continue
oc apply -f nodeset-${CELL}.yaml
done
  12. If you use a Red Hat Ceph Storage back end for Block Storage service (cinder), prepare the adopted data plane workloads: 
    for CELL in $(echo $RENAMED_CELLS); do
test -f nodeset-${CELL}.yaml || continue
oc patch osdpns/openstack-$CELL --type=merge --patch "
spec:
  services:
    - redhat
    - bootstrap
    - download-cache
    - configure-network
    - validate-network
    - install-os
    - configure-os
    - ssh-known-hosts
    - run-os
    - reboot-os
    - ceph-client
    - install-certs
    - ovn
    - neutron-metadata
    - libvirt
    - nova-$CELL
    - telemetry
  nodeTemplate:
    extraMounts:
    - extraVolType: Ceph
      volumes:
      - name: ceph
        secret:
          secretName: ceph-conf-files
      mounts:
      - name: ceph
        mountPath: "/etc/ceph"
        readOnly: true
  "
done
  13. Optional: Enable neutron-sriov-nic-agent in the OpenStackDataPlaneNodeSet CR: 
    for CELL in $(echo $RENAMED_CELLS); do
test -f nodeset-${CELL}.yaml || continue
oc patch openstackdataplanenodeset openstack-$CELL --type='json' --patch='[
{
  "op": "add",
  "path": "/spec/services/-",
  "value": "neutron-sriov"
}, {
  "op": "add",
  "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_physical_device_mappings",
  "value": "dummy_sriov_net:dummy-dev"
}, {
  "op": "add",
  "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_bandwidths",
  "value": "dummy-dev:40000000:40000000"
}, {
  "op": "add",
  "path": "/spec/nodeTemplate/ansible/ansibleVars/edpm_neutron_sriov_agent_SRIOV_NIC_resource_provider_hypervisors",
  "value": "dummy-dev:standalone.localdomain"
}]'
done
  14. Optional: Enable neutron-dhcp in the OpenStackDataPlaneNodeSet CR: 
    for CELL in $(echo $RENAMED_CELLS); do
test -f nodeset-${CELL}.yaml || continue
oc patch openstackdataplanenodeset openstack-$CELL --type='json' --patch='[
{
  "op": "add",
  "path": "/spec/services/-",
  "value": "neutron-dhcp"
}]'
done
    Note

    To use neutron-dhcp with OVN for the Bare Metal Provisioning service (ironic), you must set the disable_ovn_dhcp_for_baremetal_ports configuration option for the Networking service (neutron) to true. You can set this configuration in the NeutronAPI spec:

    ..
spec:
  serviceUser: neutron
   ...
      customServiceConfig: |
          [DEFAULT]
          dhcp_agent_notification = True
          [ovn]
          disable_ovn_dhcp_for_baremetal_ports = true
  15. Run the pre-adoption validation:
    Note
    Do not skip or disable the pre-adoption validation. Skipping this validation can cause workload downtime.
    1. Create the validation service: 
      $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
  name: pre-adoption-validation
spec:
  playbook: osp.edpm.pre_adoption_validation
EOF
    2. Create a OpenStackDataPlaneDeployment CR that runs only the validation: 
      $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: openstack-pre-adoption
spec:
  nodeSets: $NODESETS
  servicesOverride:
  - pre-adoption-validation
EOF
      Note
      If you created different migration SSH keys for different OpenStackDataPlaneService CRs, you should also define a separate OpenStackDataPlaneDeployment CR for each node set or node sets that represent a cell.
    3. When the validation is finished, confirm that the status of the Ansible EE pods is Completed: 
      $ watch oc get pod -l app=openstackansibleee
       
      $ oc logs -l app=openstackansibleee -f --max-log-requests 20
    4. Wait for the deployment to reach the Ready status: 
      $ oc wait --for condition=Ready openstackdataplanedeployment/openstack-pre-adoption --timeout=10m
      Important

      If any openstack-pre-adoption validations fail, you must reference the Ansible logs to determine which ones were unsuccessful, and then try the following troubleshooting options:

      • If the hostname validation failed, check that the hostname of the data plane node is correctly listed in the OpenStackDataPlaneNodeSet CR.
      • If the kernel argument check failed, ensure that the kernel argument configuration in the edpm_kernel_args and edpm_kernel_hugepages variables in the OpenStackDataPlaneNodeSet CR is the same as the kernel argument configuration that you used in the Red Hat OpenStack Platform (RHOSP) 17.1 node.
      • If the tuned profile check failed, ensure that the edpm_tuned_profile variable in the OpenStackDataPlaneNodeSet CR is configured to use the same profile as the one set on the RHOSP 17.1 node.
  16. Remove the remaining director services:
    1. Create an OpenStackDataPlaneService CR to clean up the data plane services you are adopting: 
      $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
  name: tripleo-cleanup
spec:
  playbook: osp.edpm.tripleo_cleanup
EOF
    2. Create the OpenStackDataPlaneDeployment CR to run the clean-up: 
      $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: tripleo-cleanup
spec:
  nodeSets: $NODESETS
  servicesOverride:
  - tripleo-cleanup
EOF
  17. When the clean-up is finished, deploy the OpenStackDataPlaneDeployment CR: 
    $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: openstack
spec:
  nodeSets: $NODESETS
EOF
    Note
    If you have other node sets to deploy, such as Networker nodes, you can add them in the nodeSets list in this step, or create separate OpenStackDataPlaneDeployment CRs later. You cannot add new node sets to an OpenStackDataPlaneDeployment CR after deployment.

Results
Copy link

  1. Confirm that all the Ansible EE pods reach a Completed status: 
    $ watch oc get pod -l app=openstackansibleee
     
    $ oc logs -l app=openstackansibleee -f --max-log-requests 20
  2. Wait for the data plane node sets to reach the Ready status: 
    for CELL in $(echo $RENAMED_CELLS); do
oc wait --for condition=Ready osdpns/openstack-$CELL --timeout=30m
done
  3. Verify that the Networking service (neutron) agents are running: 
    $ oc exec openstackclient -- openstack network agent list
+--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+
| ID                                   | Agent Type                   | Host                   | Availability Zone | Alive | State | Binary                     |
+--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+
| 174fc099-5cc9-4348-b8fc-59ed44fcfb0e | DHCP agent                   | standalone.localdomain | nova              | :-)   | UP    | neutron-dhcp-agent         |
| 10482583-2130-5b0d-958f-3430da21b929 | OVN Metadata agent           | standalone.localdomain |                   | :-)   | UP    | neutron-ovn-metadata-agent |
| a4f1b584-16f1-4937-b2b0-28102a3f6eaa | OVN Controller agent         | standalone.localdomain |                   | :-)   | UP    | ovn-controller             |
+--------------------------------------+------------------------------+------------------------+-------------------+-------+-------+----------------------------+
Note

After you remove all the services from the director cell controllers, you can decommission the cell controllers. To create new cell Compute nodes, you re-provision the decommissioned controllers as new data plane hosts and add them to the node sets of corresponding or new cells.

What to do next
Copy link

Configure data plane node sets for DCN sites
Copy link

If you are adopting a Distributed Compute Node (DCN) deployment, you must create separate OpenStackDataPlaneNodeSet custom resources (CRs) for each site. Each node set requires site-specific configuration for network subnets, OVN bridge mappings, and inter-site routes.

Before you begin
Copy link

  • You have adopted the Red Hat OpenStack Platform (RHOSP) control plane to Red Hat OpenStack Services on OpenShift (RHOSO).
  • You have configured control plane networking for your spine-leaf topology, including multi-subnet NetConfig and NetworkAttachmentDefinition CRs with routes to remote sites. For more information, see Configuring control plane networking for spine-leaf topologies.
  • You have the network configuration information for each DCN site:
    • IP addresses and hostnames for all Compute nodes
    • VLAN IDs for each service network
    • Gateway addresses for inter-site routing
  • You have identified the OVN bridge mappings (physnets) for each site.

Procedure
Copy link

  1. Define the OVN bridge mappings for each site. Each site requires a unique physnet that maps to the local provider network bridge:
    Table 1. Example OVN bridge mappings
    Site OVN bridge mapping

    Central

    leaf0:br-ex

    DCN1

    leaf1:br-ex

    DCN2

    leaf2:br-ex

  2. Create an OpenStackDataPlaneNodeSet CR for the central site Compute nodes: 
    apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneNodeSet
metadata:
  name: openstack-edpm
spec:
  tlsEnabled: false
  networkAttachments:
    - ctlplane
  preProvisioned: true
  services:
    - redhat
    - bootstrap
    - download-cache
    - configure-network
    - validate-network
    - install-os
    - configure-os
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ovn
    - neutron-metadata
    - libvirt
    - nova-cell1
    - telemetry
  nodeTemplate:
    ansibleSSHPrivateKeySecret: dataplane-adoption-secret
    ansible:
      ansibleUser: tripleo-admin
      ansibleVars:
        edpm_ovn_bridge_mappings: ["leaf0:br-ex"]
        edpm_ovn_bridge: br-int
        edpm_ovn_encap_type: geneve
        # Network configuration template for central site
        edpm_network_config_template: |
          ---
          network_config:
          - type: ovs_bridge
            name: {{ neutron_physical_bridge_name }}
            use_dhcp: false
            dns_servers: {{ ctlplane_dns_nameservers }}
            addresses:
            - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
            routes: {{ ctlplane_host_routes }}
            members:
            - type: interface
              name: nic1
              primary: true
          {% for network in nodeset_networks %}
            - type: vlan
              vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
              addresses:
              - ip_netmask:
                  {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
          {% endfor %}
  nodes:
    compute-0:
      hostName: compute-0.example.com
      ansible:
        ansibleHost: compute-0.example.com
      networks:
      - defaultRoute: true
        fixedIP: 192.168.122.100
        name: ctlplane
        subnetName: subnet1
      - name: internalapi
        subnetName: subnet1
      - name: storage
        subnetName: subnet1
      - name: tenant
        subnetName: subnet1
    • The OVN bridge mapping uses the central site physnet leaf0.
    • Central site nodes reference subnet1 for all networks.
  3. Create an OpenStackDataPlaneNodeSet CR for DCN1 edge site Compute nodes. You must add inter-site routes to the network configuration template: 
    apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneNodeSet
metadata:
  name: openstack-edpm-dcn1
spec:
  tlsEnabled: false
  networkAttachments:
    - ctlplane
  preProvisioned: true
  services:
    - redhat
    - bootstrap
    - download-cache
    - configure-network
    - validate-network
    - install-os
    - configure-os
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ovn
    - neutron-metadata
    - libvirt
    - nova-cell1
    - telemetry
  nodeTemplate:
    ansibleSSHPrivateKeySecret: dataplane-adoption-secret
    ansible:
      ansibleUser: tripleo-admin
      ansibleVars:
        edpm_ovn_bridge_mappings: ["leaf1:br-ex"]
        edpm_ovn_bridge: br-int
        edpm_ovn_encap_type: geneve
        # Network configuration template for DCN1 site with inter-site routes
        edpm_network_config_template: |
          ---
          network_config:
          - type: ovs_bridge
            name: {{ neutron_physical_bridge_name }}
            use_dhcp: false
            dns_servers: {{ ctlplane_dns_nameservers }}
            addresses:
            - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
            routes:  #
              {{ ctlplane_host_routes }}
              - ip_netmask: 192.168.122.0/24
                next_hop: 192.168.133.1
              - ip_netmask: 192.168.144.0/24
                next_hop: 192.168.133.1
            members:
            - type: interface
              name: nic1
              primary: true
          {% for network in nodeset_networks %}
            - type: vlan
              vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
              addresses:
              - ip_netmask:
                  {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
              routes:
                {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
                {% if network == 'internalapi' %}
                - ip_netmask: 172.17.0.0/24
                  next_hop: 172.17.10.1
                - ip_netmask: 172.17.20.0/24
                  next_hop: 172.17.10.1
                {% endif %}
                {% if network == 'storage' %}
                - ip_netmask: 172.18.0.0/24
                  next_hop: 172.18.10.1
                - ip_netmask: 172.18.20.0/24
                  next_hop: 172.18.10.1
                {% endif %}
                {% if network == 'tenant' %}
                - ip_netmask: 172.19.0.0/24
                  next_hop: 172.19.10.1
                - ip_netmask: 172.19.20.0/24
                  next_hop: 172.19.10.1
                {% endif %}
          {% endfor %}
  nodes:
    dcn1-compute-0:
      hostName: dcn1-compute-0.example.com
      ansible:
        ansibleHost: dcn1-compute-0.example.com
      networks:
      - defaultRoute: true
        fixedIP: 192.168.133.100
        name: ctlplane
        subnetName: ctlplanedcn1
      - name: internalapi
        subnetName: internalapidcn1
      - name: storage
        subnetName: storagedcn1
      - name: tenant
        subnetName: tenantdcn1
    • DCN1 uses the leaf1 physnet for its OVN bridge mapping under spec:nodeTemplate:ansible:ansibleVars:edpm_ovn_bridge_mappings.
    • Inter-site routes must be added to the network configuration template. These routes enable DCN1 compute nodes to reach the central site (192.168.122.0/24) and other DCN sites (192.168.144.0/24 for DCN2). Similar routes are added for each service network (internalapi, storage, tenant).
    • DCN1 nodes reference site-specific subnet names such as ctlplanedcn1 and internalapidcn1. These subnet names must match those defined in the NetConfig CR.
  4. Repeat step 3 for all other DCN sites. Adjust site specific parameters:
    • The node set name, for example: openstack-edpm-dcn2
    • The OVN bridge mapping, for example: leaf2:br-ex
    • The subnet names, for example: ctlplanedcn2, and internalapidcn2
    • The inter-site routes. The routes from DCN2 should point to the central site subnets and the DCN1 site subnets.
    • The compute node definitions with site-appropriate IP addresses.
  5. Deploy all node sets by creating an OpenStackDataPlaneDeployment CR: 
    apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: openstack-edpm-deployment
spec:
  nodeSets:
    - openstack-edpm
    - openstack-edpm-dcn1
    - openstack-edpm-dcn2
    Note

    All node sets can be deployed in parallel once the control plane adoption is complete.
  6. Wait for the deployment to complete: 
    $ oc wait --for condition=Ready openstackdataplanedeployment/openstack-edpm-deployment --timeout=40m

Results
Copy link

  1. Verify that all node sets reach the Ready status: 
    $ oc get openstackdataplanenodeset
NAME                  STATUS   MESSAGE
openstack-edpm        True     Ready
openstack-edpm-dcn1   True     Ready
openstack-edpm-dcn2   True     Ready
  2. Verify that Compute services are running across all sites. Ensure that all nova-compute services show State=up for nodes in all availability zones: 
    $ oc exec openstackclient -- openstack compute service list
  3. Verify inter-site connectivity by checking routes on a DCN Compute node: 
    $ ssh dcn1-compute-0 ip route show | grep 172.17.0
172.17.0.0/24 via 172.17.10.1 dev internalapi
  4. Test that DCN Compute nodes can reach the control plane: 
    $ ssh dcn1-compute-0 ping -c 3 172.17.0.30

    Replace 172.17.0.30 with an IP address of a control plane service on the internalapi network.

Adopt Compute services with DCN back ends
Copy link

Configure the edge Compute service (nova) nodes at each site with the Red Hat Ceph Storage connection details and Image Service (glance) endpoint for their local site.

Before you begin
Copy link

About this task
Copy link

In a Distributed Compute Node (DCN) deployment where Image service and Block Storage service (cinder) services run on edge Compute nodes, each site has its own Red Hat Ceph Storage cluster. The Image service has a separate API endpoint at each site. Each site’s OpenStackDataPlaneNodeSet custom resource (CR) must use a different OpenStackDataPlaneService CR that points to the correct Image service.

In a DCN deployment, all node sets belong to a single Compute service cell. The central site and each edge site are separate OpenStackDataPlaneNodeSet resources within that cell. The per-site OpenStackDataPlaneService resources deliver different Red Hat Ceph Storage and Image service configurations to each node set while sharing the same cell-level Compute service configuration.

Procedure
Copy link

  1. Set the cell name variable. In a DCN deployment, all node sets belong to a single cell: 
    $ DEFAULT_CELL_NAME="cell1"
  2. Retrieve the fsid for each Red Hat Ceph Storage cluster and store them in shell variables: 
    $ CEPH_FSID_CENTRAL=$(oc get secret ceph-conf-central -o json | jq -r '.data."<central.conf>"' | base64 -d | awk '/fsid/{print $3}')
$ CEPH_FSID_DCN1=$(oc get secret ceph-conf-dcn1 -o json | jq -r '.data."<dcn1.conf>"' | base64 -d | awk '/fsid/{print $3}')
$ CEPH_FSID_DCN2=$(oc get secret ceph-conf-dcn2 -o json | jq -r '.data."<dcn2.conf>"' | base64 -d | awk '/fsid/{print $3}')

    where:

    <central.conf>
    Specifies the name of the Red Hat Ceph Storage configuration file for the central site in the ceph-conf-central secret.
    <dcn1.conf>
    Specifies the name of the Red Hat Ceph Storage configuration file for an edge site in the ceph-conf-dcn1 secret.
    <dcn2.conf>
    Specifies the name of the Red Hat Ceph Storage configuration file for an additional edge site in the ceph-conf-dcn2 secret.
  3. Create a ConfigMap for each site. Each ConfigMap contains the Red Hat Ceph Storage and Image service configuration specific to that site.

    The following example creates ConfigMap resources for a central site and two edge sites.

    1. Create the ConfigMap for the central site: 
      $ oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: nova-ceph-central
data:
  99-nova-compute-cells-workarounds.conf: |
    [workarounds]
    disable_compute_service_check_for_ffu=true
  03-ceph-nova.conf: |
    [libvirt]
    images_type=rbd
    images_rbd_pool=vms
    images_rbd_ceph_conf=/etc/ceph/central.conf
    images_rbd_glance_store_name=central
    images_rbd_glance_copy_poll_interval=15
    images_rbd_glance_copy_timeout=600
    rbd_user=openstack
    rbd_secret_uuid=${CEPH_FSID_CENTRAL}
    [glance]
    endpoint_override = http://glance-central-internal.openstack.svc:9292
    valid_interfaces = internal
    [cinder]
    cross_az_attach = False
    catalog_info = volumev3:cinderv3:internalURL
EOF

      Each ConfigMap contains three configuration sections:

      • [libvirt] points to the local Red Hat Ceph Storage cluster configuration and uses the local fsid as the rbd_secret_uuid.
      • [glance] uses endpoint_override to direct Image service requests to the local Image service API endpoint instead of the endpoint that is registered in the Identity service catalog. The examples use http:// for the Image service endpoints. If your Red Hat OpenStack Platform deployment uses TLS for internal endpoints, use https:// instead, and ensure that you have completed the TLS migration. For more information, see Migrating TLS-e to the RHOSO deployment.
      • [cinder] sets cross_az_attach = False to prevent volumes from being attached to instances in a different availability zone.
    2. Create the ConfigMap for the first edge site: 
      $ oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: nova-ceph-dcn1
data:
  99-nova-compute-cells-workarounds.conf: |
    [workarounds]
    disable_compute_service_check_for_ffu=true
  03-ceph-nova.conf: |
    [libvirt]
    images_type=rbd
    images_rbd_pool=vms
    images_rbd_ceph_conf=/etc/ceph/dcn1.conf
    images_rbd_glance_store_name=dcn1
    images_rbd_glance_copy_poll_interval=15
    images_rbd_glance_copy_timeout=600
    rbd_user=openstack
    rbd_secret_uuid=${CEPH_FSID_DCN1}
    [glance]
    endpoint_override = http://glance-dcn1-internal.openstack.svc:9292
    valid_interfaces = internal
    [cinder]
    cross_az_attach = False
    catalog_info = volumev3:cinderv3:internalURL
EOF
    3. Create the ConfigMap for the second edge site: 
      $ oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: nova-ceph-dcn2
data:
  99-nova-compute-cells-workarounds.conf: |
    [workarounds]
    disable_compute_service_check_for_ffu=true
  03-ceph-nova.conf: |
    [libvirt]
    images_type=rbd
    images_rbd_pool=vms
    images_rbd_ceph_conf=/etc/ceph/dcn2.conf
    images_rbd_glance_store_name=dcn2
    images_rbd_glance_copy_poll_interval=15
    images_rbd_glance_copy_timeout=600
    rbd_user=openstack
    rbd_secret_uuid=${CEPH_FSID_DCN2}
    [glance]
    endpoint_override = http://glance-dcn2-internal.openstack.svc:9292
    valid_interfaces = internal
    [cinder]
    cross_az_attach = False
    catalog_info = volumev3:cinderv3:internalURL
EOF
      Important

      The endpoint_override in the [glance] section is different for each site. This setting directs the Compute service to contact the local Image service API instead of the central endpoint registered in the Identity service catalog. Without this setting, all Compute nodes contact the central Image service, and image data is transferred across the WAN instead of read from the local Red Hat Ceph Storage cluster.

      • Central Compute nodes use glance-central-internal.openstack.svc
      • DCN1 Compute nodes use glance-dcn1-internal.openstack.svc
      • DCN2 Compute nodes use glance-dcn2-internal.openstack.svc

      These endpoint names correspond to the GlanceAPI instances that are created when you adopt the Image service with DCN back ends.
  4. Create a per-site OpenStackDataPlaneService CR for each site. Each service references the site-specific ConfigMap that you created in the previous step: 
    $ oc apply -f - <<EOF
---
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
  name: nova-custom-ceph-central
spec:
  dataSources:
    - configMapRef:
        name: nova-ceph-central
    - secretRef:
        name: nova-${DEFAULT_CELL_NAME}-compute-config
    - secretRef:
        name: nova-migration-ssh-key
  playbook: osp.edpm.nova
  caCerts: combined-ca-bundle
  edpmServiceType: nova
  containerImageFields:
  - NovaComputeImage
  - EdpmIscsidImage
---
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
  name: nova-custom-ceph-dcn1
spec:
  dataSources:
    - configMapRef:
        name: nova-ceph-dcn1
    - secretRef:
        name: nova-${DEFAULT_CELL_NAME}-compute-config
    - secretRef:
        name: nova-${DEFAULT_CELL_NAME}-metadata-neutron-config
    - secretRef:
        name: nova-migration-ssh-key
  playbook: osp.edpm.nova
  caCerts: combined-ca-bundle
  edpmServiceType: nova
  containerImageFields:
  - NovaComputeImage
  - EdpmIscsidImage
---
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
  name: nova-custom-ceph-dcn2
spec:
  dataSources:
    - configMapRef:
        name: nova-ceph-dcn2
    - secretRef:
        name: nova-${DEFAULT_CELL_NAME}-compute-config
    - secretRef:
        name: nova-${DEFAULT_CELL_NAME}-metadata-neutron-config
    - secretRef:
        name: nova-migration-ssh-key
  playbook: osp.edpm.nova
  caCerts: combined-ca-bundle
  edpmServiceType: nova
  containerImageFields:
  - NovaComputeImage
  - EdpmIscsidImage
EOF
    Note

    All OpenStackDataPlaneService CRs reference the same cell secret (nova-cell1-compute-config) because all node sets belong to a single cell. The per-site ConfigMap is what differentiates the Red Hat Ceph Storage and Image service configuration for each site.
  5. When you create the OpenStackDataPlaneNodeSet CR for each site, reference the per-site service in the services list instead of nova-$CELL. For example:
    • The central node set uses nova-custom-ceph-central in its services list.
    • The DCN1 node set uses nova-custom-ceph-dcn1 in its services list.
    • The DCN2 node set uses nova-custom-ceph-dcn2 in its services list.
      1. If you have already created the OpenStackDataPlaneNodeSet CRs with the default nova-$CELL service, patch each node set to use the per-site service. The following example patches the central node set: 
        $ oc patch osdpns/openstack-${DEFAULT_CELL_NAME} --type=merge --patch "
spec:
  services:
    - bootstrap
    - download-cache
    - configure-network
    - validate-network
    - install-os
    - configure-os
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ceph-client
    - ovn
    - neutron-metadata
    - libvirt
    - nova-custom-ceph-central
  nodeTemplate:
    extraMounts:
    - extraVolType: Ceph
      volumes:
      - name: ceph
        secret:
          secretName: ceph-conf-central
      mounts:
      - name: ceph
        mountPath: "/etc/ceph"
        readOnly: true
"
      2. Patch each DCN edge node set with the same services list, replacing ovn with ovn-dcn and nova-custom-ceph-central with the per-site service name. You must include the ceph-client service so that the Red Hat Ceph Storage configuration files from the per-site secret are deployed into the Compute service containers on the edge nodes. Without ceph-client, the /etc/ceph/ directory inside the Compute service container is empty and instances fail to launch with a RADOS object not found (error calling conf_read_file) error.

        For example, for the DCN1 node set named dcn1:

        $ oc patch osdpns/dcn1 --type=merge --patch "
spec:
  services:
    - bootstrap
    - download-cache
    - configure-network
    - validate-network
    - install-os
    - configure-os
    - ssh-known-hosts
    - run-os
    - reboot-os
    - install-certs
    - ceph-client
    - ovn-dcn
    - neutron-metadata
    - libvirt
    - nova-custom-ceph-dcn1
  nodeTemplate:
    extraMounts:
    - extraVolType: Ceph
      volumes:
      - name: ceph
        secret:
          secretName: ceph-conf-dcn1
      mounts:
      - name: ceph
        mountPath: "/etc/ceph"
        readOnly: true
"

        Repeat this step for each additional edge site, replacing dcn1 and nova-custom-ceph-dcn1 with the appropriate site name, for example, dcn2 and nova-custom-ceph-dcn2.

Perform a fast-forward upgrade on Compute services
Copy link

You must upgrade the Compute services from Red Hat OpenStack Platform 17.1 to Red Hat OpenStack Services on OpenShift (RHOSO) 18.0 on the control plane and data plane.

Before you begin
Copy link

  • You have defined the shell variables necessary to apply the fast-forward upgrade commands for each Compute service cell. 
    DEFAULT_CELL_NAME="cell1"
RENAMED_CELLS="$DEFAULT_CELL_NAME"

declare -A PODIFIED_DB_ROOT_PASSWORD
for CELL in $(echo "super $RENAMED_CELLS"); do
  PODIFIED_DB_ROOT_PASSWORD[$CELL]=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
done
  • You have completed the steps in Adopting Compute services to the RHOSO data plane.

About this task
Copy link

Complete the following tasks:

  • Update the cell1 Compute data plane services version.
  • Remove pre-fast-forward upgrade workarounds from the Compute control plane services and Compute data plane services.
  • Run Compute database online migrations to update live data.

Procedure
Copy link

  1. Wait for the Compute service data plane services version to update for all the cells: 
    for CELL in $(echo $RENAMED_CELLS); do
oc exec openstack-$CELL-galera-0 -c galera -- mysql -rs -uroot -p"${PODIFIED_DB_ROOT_PASSWORD[$CELL]}" \
  -e "select a.version from nova_${CELL}.services a join nova_${CELL}.services b where a.version!=b.version and a.binary='nova-compute' and a.deleted=0;"
done
    Note

    The query returns an empty result when the update is completed. No downtime is expected for virtual machine (VM) workloads.

    Review any errors in the nova Compute agent logs on the data plane, and the nova-conductor journal records on the control plane.
  2. Patch the OpenStackControlPlane CR to remove the pre-fast-forward upgrade workarounds from the Compute control plane services: 
    $ rm -f celltemplates
$ for CELL in $(echo $RENAMED_CELLS); do
$ cat >> celltemplates << EOF
        ${CELL}:
          metadataServiceTemplate:
            customServiceConfig: |
              [workarounds]
              disable_compute_service_check_for_ffu=false
          conductorServiceTemplate:
            customServiceConfig: |
              [workarounds]
              disable_compute_service_check_for_ffu=false
EOF
done

$ cat > oscp-patch.yaml << EOF
spec:
  nova:
    template:
      apiServiceTemplate:
        customServiceConfig: |
          [workarounds]
          disable_compute_service_check_for_ffu=false
      metadataServiceTemplate:
        customServiceConfig: |
          [workarounds]
          disable_compute_service_check_for_ffu=false
      schedulerServiceTemplate:
        customServiceConfig: |
          [workarounds]
          disable_compute_service_check_for_ffu=false
      cellTemplates:
        cell0:
          conductorServiceTemplate:
            customServiceConfig: |
              [workarounds]
              disable_compute_service_check_for_ffu=false
EOF
$ cat celltemplates >> oscp-patch.yaml
    • If you are adopting the Compute service with the Bare Metal Provisioning service (ironic), append the following novaComputeTemplates in the cell<X> section of the Compute service CR patch: 
              cell<X>:
          novaComputeTemplates:
            <hostname>:
              customServiceConfig: |
                [DEFAULT]
                host = <hostname>
                [workarounds]
                disable_compute_service_check_for_ffu=true
              computeDriver: ironic.IronicDriver
        ...

      where:

      <hostname>
      Specifies the hostname of the node that is running the ironic Compute driver in the source cloud of cell<X>.
  3. Apply the patch file: 
    $ oc patch openstackcontrolplane openstack --type=merge --patch-file=oscp-patch.yaml
  4. Wait until the Compute control plane services CRs are ready: 
    $ oc wait --for condition=Ready --timeout=300s Nova/nova
  5. Remove the pre-fast-forward upgrade workarounds from the Compute data plane services: 
    $ oc patch cm nova-cells-global-config --type=json -p='[{"op": "replace", "path": "/data/99-nova-compute-cells-workarounds.conf", "value": "[workarounds]\n"}]'
$ for CELL in $(echo $RENAMED_CELLS); do
$ oc get Openstackdataplanenodeset openstack-${CELL} || continue
$ oc apply -f - <<EOF
---
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: openstack-nova-compute-ffu-$CELL
spec:
  nodeSets:
    - openstack-${CELL}
  servicesOverride:
    - nova-${CELL}
backoffLimit: 3
EOF
done
  6. Wait for the Compute data plane services to be ready for all the cells: 
    $ oc wait --for condition=Ready openstackdataplanedeployments --all --timeout=5m
  7. Run Compute database online migrations to complete the upgrade: 
    $ oc exec -it nova-cell0-conductor-0 -- nova-manage db online_data_migrations
$ for CELL in $(echo $RENAMED_CELLS); do
$ oc exec -it nova-${CELL}-conductor-0 -- nova-manage db online_data_migrations
done
  8. Discover the Compute hosts in the cells: 
    $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
  9. If you have a test VM that is not a production workload, complete the following verification steps:
    1. Verify if the existing test VM instance is running: 
      ${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test 2>&1 || echo FAIL
    2. Verify if the Compute services can stop the existing test VM instance: 
      ${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test ACTIVE" && ${BASH_ALIASES[openstack]} server stop test || echo PASS
${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test SHUTOFF" || echo FAIL
${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test 2>&1 || echo PASS
    3. Verify if the Compute services can start the existing test VM instance: 
      ${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test SHUTOFF" && ${BASH_ALIASES[openstack]} server start test || echo PASS
${BASH_ALIASES[openstack]} server list -c Name -c Status -f value | grep -qF "test ACTIVE" && \
  ${BASH_ALIASES[openstack]} server --os-compute-api-version 2.48 show --diagnostics test --fit-width -f json | jq -r '.state' | grep running || echo FAIL

What to do next
Copy link

After the data plane adoption, the Compute hosts continue to run Red Hat Enterprise Linux (RHEL) 8.4. To take advantage of RHEL 9.2, perform a minor update procedure after finishing the adoption procedure.

Adopt Networker services to the RHOSO data plane
Copy link

Adopt Networker services to the Red Hat OpenStack Services on OpenShift (RHOSO) data plane by creating a OpenStackDataPlaneNodeSet custom resource (CR) that includes the services to run on the Networker nodes. The Networker services could be running on Controller nodes or dedicated Networker nodes.

Before you begin
Copy link

  • Define the shell variable. Based on above agent list output, controller-0, controller-1, controller-2 are our target hosts. If you have both Controller and Networker nodes running networker services then add all those hosts below. 
    declare -A networkers
networkers+=(
  ["controller-0.localdomain"]="192.168.122.100"
  ["controller-1.localdomain"]="192.168.122.101"
  ["controller-2.localdomain"]="192.168.122.102"
  # ...
)
    • Replace ["<node-name>"]="192.168.122.100" with the name and IP address of the corresponding Networker or Controller node in your environment.

About this task
Copy link

Tip

By definition, any node that has set enable-chassis-as-gw is considered a Networker node. After the adoption process, these nodes continue to be Networker nodes.

 Note

If you want to reclaim the hardware from your Red Hat OpenStack Platform Controller or Networker nodes after adoption, you can migrate the OVN gateway services to Red Hat OpenShift Container Platform (RHOCP) worker nodes instead of keeping them on the original nodes. When migrating RHOSO worker nodes, expect disruption on the data plane in centralized or SNAT traffic as the router gateway ports move from one node to another.

You can implement the following options if they apply to your environment:

  • Depending on your topology, you might need to run the neutron-metadata service on the nodes, specifically when you want to serve metadata to SR-IOV ports that are hosted on Compute nodes.
  • If you want to continue running OVN gateway services on Networker nodes, keep ovn service in the list to deploy.
  • Optional: You can run the neutron-dhcp service on your Networker nodes instead of your Compute nodes. You might not need to use neutron-dhcp with OVN, unless your deployment uses DHCP relays, or advanced DHCP options that are supported by dnsmasq but not by the OVN DHCP implementation.

Adopt each Controller or Networker node in your existing Red Hat OpenStack Platform deployment to the Red Hat OpenStack Services on OpenShift (RHOSO) when your node is set as an OVN chassis gateway. Any node with parameter set to enable-chassis-as-gw is considered OVN gateway chassis. In this case, such nodes will become data plane networker nodes after adoption.

  1. Check for the nodes where OVN Controller Gateway agent agents are running. The list of agents varies depending on the services you enabled: 
    $ oc exec openstackclient -- openstack network agent list
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
| ID                                   | Agent Type                   | Host                     | Availability Zone | Alive | State | Binary                     |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
| e5075ee0-9dd9-4f0a-a42a-6bbdf1a6111c | OVN Controller Gateway agent | controller-0.localdomain |                   | XXX   | UP    | ovn-controller             |
| f3112349-054c-403a-b00a-e219238192b8 | OVN Controller agent         | compute-0.localdomain    |                   | XXX   | UP    | ovn-controller             |
| af9dae2d-1c1c-55a8-a743-f84719f6406d | OVN Metadata agent           | compute-0.localdomain    |                   | XXX   | UP    | neutron-ovn-metadata-agent |
| 51a11df8-a66e-47a2-aec0-52eb8589626c | OVN Controller Gateway agent | controller-1.localdomain |                   | XXX   | UP    | ovn-controller             |
| bb817e5e-7832-410a-9e67-934dac8c602f | OVN Controller Gateway agent | controller-2.localdomain |                   | XXX   | UP    | ovn-controller             |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+

Procedure
Copy link

  1. Deploy the OpenStackDataPlaneNodeSet CR for your nodes:
    Note
    You can reuse most of the nodeTemplate section from the OpenStackDataPlaneNodeSet CR that is designated for your Compute nodes. You can omit some of the variables because of the limited set of services that are running on the Networker nodes.
     
    $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneNodeSet
metadata:
  name: openstack-networker
spec:
  tlsEnabled: false
  networkAttachments:
      - ctlplane
  preProvisioned: true
  services:
    - redhat
    - bootstrap
    - download-cache
    - configure-network
    - validate-network
    - install-os
    - configure-os
    - ssh-known-hosts
    - run-os
    - install-certs
    - ovn
  env:
    - name: ANSIBLE_CALLBACKS_ENABLED
      value: "profile_tasks"
    - name: ANSIBLE_FORCE_COLOR
      value: "True"
  nodes:
    controller-0:
      hostName: controller-0
      ansible:
        ansibleHost: ${networkers[controller-0.localdomain]}
      networks:
      - defaultRoute: true
        fixedIP: ${networkers[controller-0.localdomain]}
        name: ctlplane
        subnetName: subnet1
      - name: internalapi
        subnetName: subnet1
      - name: storage
        subnetName: subnet1
      - name: tenant
        subnetName: subnet1
    controller-1:
      hostName: controller-1
      ansible:
        ansibleHost: ${networkers[controller-1.localdomain]}
      networks:
      - defaultRoute: true
        fixedIP: ${networkers[controller-1.localdomain]}
        name: ctlplane
        subnetName: subnet1
      - name: internalapi
        subnetName: subnet1
      - name: storage
        subnetName: subnet1
      - name: tenant
        subnetName: subnet1
    controller-2:
      hostName: controller-2
      ansible:
        ansibleHost: ${networkers[controller-2.localdomain]}
      networks:
      - defaultRoute: true
        fixedIP: ${networkers[controller-2.localdomain]}
        name: ctlplane
        subnetName: subnet1
      - name: internalapi
        subnetName: subnet1
      - name: storage
        subnetName: subnet1
      - name: tenant
        subnetName: subnet1
  nodeTemplate:
    ansibleSSHPrivateKeySecret: dataplane-adoption-secret
    ansible:
      ansibleUser: tripleo-admin
      ansibleVarsFrom:
      - secretRef:
          name: subscription-manager
      - secretRef:
          name: redhat-registry
      ansibleVars:
        rhc_release: 9.2
        rhc_repositories:
            - {name: "*", state: disabled}
            - {name: "<repository_name1>", state: enabled}
            - {name: "<repository_name2>", state: enabled}
            - {name: "<repository_name3>", state: enabled}
            - {name: "<repository_name4>", state: enabled}
            - {name: "<repository_name5>", state: enabled}
            - {name: "<repository_name6>", state: enabled}
        edpm_bootstrap_release_version_package: []
        # edpm_network_config
        # Default nic config template for a EDPM node
        # These vars are edpm_network_config role vars
        edpm_network_config_template: |
           ---
           {% set mtu_list = [ctlplane_mtu] %}
           {% for network in nodeset_networks %}
           {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
           {%- endfor %}
           {% set min_viable_mtu = mtu_list | max %}
           network_config:
           - type: ovs_bridge
             name: {{ neutron_physical_bridge_name }}
             mtu: {{ min_viable_mtu }}
             use_dhcp: false
             dns_servers: {{ ctlplane_dns_nameservers }}
             domain: {{ dns_search_domains }}
             addresses:
             - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_cidr }}
             routes: {{ ctlplane_host_routes }}
             members:
             - type: interface
               name: nic1
               mtu: {{ min_viable_mtu }}
               # force the MAC address of the bridge to this interface
               primary: true
           {% for network in nodeset_networks %}
             - type: vlan
               mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
               vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
               addresses:
               - ip_netmask:
                   {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
           {% endfor %}
        edpm_network_config_nmstate: false
        edpm_network_config_hide_sensitive_logs: false
        #
        # These vars are for the network config templates themselves and are
        # considered EDPM network defaults.
        neutron_physical_bridge_name: br-ctlplane
        neutron_public_interface_name: eth0

        # edpm_nodes_validation
        edpm_nodes_validation_validate_controllers_icmp: false
        edpm_nodes_validation_validate_gateway_icmp: false

        # edpm ovn-controller configuration
        edpm_ovn_bridge_mappings: [<"bridge_mappings">]
        edpm_ovn_bridge: br-int
        edpm_ovn_encap_type: geneve
        ovn_monitor_all: true
        edpm_ovn_remote_probe_interval: 60000
        edpm_ovn_ofctrl_wait_before_clear: 8000

        # serve as a OVN gateway
        edpm_enable_chassis_gw: true

        timesync_ntp_servers:
        - hostname: clock.redhat.com
        - hostname: clock2.redhat.com


        gather_facts: false
        enable_debug: false
        # edpm firewall, change the allowed CIDR if needed
        edpm_sshd_configure_firewall: true
        edpm_sshd_allowed_ranges: ['192.168.122.0/24']
        # SELinux module
        edpm_selinux_mode: enforcing

        # Do not attempt OVS major upgrades here
        edpm_ovs_packages:
        - openvswitch3.3
EOF
    • spec.tlsEnabled specifies whether TLS Everywhere is enabled. If TLS is enabled, change spec:tlsEnabled to true.
    • edpm_ovn_bridge_mappings: Replace [<"bridge_mappings">] with the bridge mapping values that you used in your Red Hat OpenStack Platform 17.1 deployment, for example, ["datacentre:br-ctlplane"].
    • edpm_enable_chassis_gw specifies whether to run ovn-controller in gateway mode.
    • <repository_name1>, <repository_name2>, <repository_name3>, <repository_name4>, <repository_name5>, <repository_name6> specifies the name of the repository to enable. For more information about which repositories are required, see Required repositories for Red Hat OpenStack Services on OpenShift 18.0.
  2. Ensure that you use the same ovn-controller settings in the OpenStackDataPlaneNodeSet CR that you used in the Networker nodes before adoption. This configuration is stored in the external_ids column in the Open_vSwitch table in the Open vSwitch database: 
    ovs-vsctl list Open .
...
external_ids        : {hostname=controller-0.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
...
    • Replace <bridge_mappings> with the value of the bridge mappings in your configuration, for example, "datacentre:br-ctlplane".
  3. Optional: Enable neutron-metadata in the OpenStackDataPlaneNodeSet CR: 
    $ oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[
  {
    "op": "add",
    "path": "/spec/services/-",
    "value": "neutron-metadata"
  }]'
    • Replace <networker_CR_name> with the name of the CR that you deployed for your Networker nodes, for example, openstack-networker.
  4. Optional: Enable neutron-dhcp in the OpenStackDataPlaneNodeSet CR: 
    $ oc patch openstackdataplanenodeset <networker_CR_name> --type='json' --patch='[
  {
    "op": "add",
    "path": "/spec/services/-",
    "value": "neutron-dhcp"
  }]'
  5. Run the pre-adoption-validation service for Networker nodes:
    1. Create a OpenStackDataPlaneDeployment CR that runs only the validation: 
      $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: openstack-pre-adoption-networker
spec:
  nodeSets:
  - openstack-networker
  servicesOverride:
  - pre-adoption-validation
EOF
    2. When the validation is finished, confirm that the status of the Ansible EE pods is Completed: 
      $ watch oc get pod -l app=openstackansibleee
       
      $ oc logs -l app=openstackansibleee -f --max-log-requests 20
    3. Wait for the deployment to reach the Ready status: 
      $ oc wait --for condition=Ready openstackdataplanedeployment/openstack-pre-adoption-networker --timeout=10m
  6. Deploy the OpenStackDataPlaneDeployment CR for Networker nodes: 
    $ oc apply -f - <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: openstack-networker
spec:
  nodeSets:
  - openstack-networker
EOF
    Note
    Alternatively, you can include the Networker node set in the nodeSets list before you deploy the main OpenStackDataPlaneDeployment CR. You cannot add new node sets to the OpenStackDataPlaneDeployment CR after deployment.
  7. Clean up any Networking service (neutron) agents that are no longer running.
    Note
    In some cases, agents from the old data plane that are replaced or retired remain in RHOSO. The function these agents provided might be provided by a new agent that is running in RHOSO, or the function might be replaced by other components. For example, DHCP agents might no longer be needed, since OVN DHCP in RHOSO can provide this function.
    1. List the agents: 
      $ oc exec openstackclient -- openstack network agent list
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
| ID                                   | Agent Type                   | Host                     | Availability Zone | Alive | State | Binary                     |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
| e5075ee0-9dd9-4f0a-a42a-6bbdf1a6111c | OVN Controller Gateway agent | controller-0.localdomain |                   | :-)   | UP    | ovn-controller             |
| 856960f0-5530-46c7-a331-6eadcba362da | DHCP agent                   | controller-1.localdomain | nova              | XXX   | UP    | neutron-dhcp-agent         |
| 8bd22720-789f-45b8-8d7d-006dee862bf9 | DHCP agent                   | controller-2.localdomain | nova              | XXX   | UP    | neutron-dhcp-agent         |
| e584e00d-be4c-4e98-a11a-4ecd87d21be7 | DHCP agent                   | controller-0.localdomain | nova              | XXX   | UP    | neutron-dhcp-agent         |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
    2. If any agent in the list shows XXX in the Alive field, verify the Host and Agent Type, if the functions of this agent is no longer required, and the agent has been permanently stopped on the Red Hat OpenStack Platform host. Then, delete the agent: 
      $ oc exec openstackclient -- openstack network agent delete <agent_id>
      • Replace <agent_id> with the ID of the agent to delete, for example, 856960f0-5530-46c7-a331-6eadcba362da.

Results
Copy link

  1. Confirm that all the Ansible EE pods reach a Completed status: 
    $ watch oc get pod -l app=openstackansibleee
     
    $ oc logs -l app=openstackansibleee -f --max-log-requests 20
  2. Wait for the data plane node set to reach the Ready status: 
    $ oc wait --for condition=Ready osdpns/<networker_CR_name> --timeout=30m
    • Replace <networker_CR_name> with the name of the CR that you deployed for your Networker nodes, for example, openstack-networker.
  3. Verify that the Networking service (neutron) agents are running. The list of agents varies depending on the services you enabled: 
    $ oc exec openstackclient -- openstack network agent list
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
| ID                                   | Agent Type                   | Host                     | Availability Zone | Alive | State | Binary                     |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+
| e5075ee0-9dd9-4f0a-a42a-6bbdf1a6111c | OVN Controller Gateway agent | controller-0.localdomain |                   | :-)   | UP    | ovn-controller             |
| f3112349-054c-403a-b00a-e219238192b8 | OVN Controller agent         | compute-0.localdomain    |                   | :-)   | UP    | ovn-controller             |
| af9dae2d-1c1c-55a8-a743-f84719f6406d | OVN Metadata agent           | compute-0.localdomain    |                   | :-)   | UP    | neutron-ovn-metadata-agent |
| 51a11df8-a66e-47a2-aec0-52eb8589626c | OVN Controller Gateway agent | controller-1.localdomain |                   | :-)   | UP    | ovn-controller             |
| bb817e5e-7832-410a-9e67-934dac8c602f | OVN Controller Gateway agent | controller-2.localdomain |                   | :-)   | UP    | ovn-controller             |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------------------+

Migrate OVN gateway services to RHOCP worker nodes
Copy link

Migrate OVN gateway services from Networker nodes to Red Hat OpenShift Container Platform (RHOCP) worker nodes to reclaim and repurpose hardware from your Red Hat OpenStack Platform (RHOSP) deployment.

Before you begin
Copy link

  • You have completed the RHOSO adoption procedure and your environment is functional.
  • You have dedicated network interface cards (NICs) available on RHOCP worker nodes that are not shared with any RHOSP or RHOCP networks. If you want to configure bonding for the external bridge, you need multiple dedicated NICs.
  • The switch ports for these dedicated NICs are configured with the same settings as the ports on the old Controller or Networker nodes, such as VLAN configuration.
  • The old Controller or Networker nodes are not running any other non-networking services after adoption, such as Object Storage (swift) or Block Storage (cinder).
  • If neutron-dhcp is running to provide DHCP to bare-metal nodes in IPv6 deployments, or if neutron-ovn-metadata is running to provide metadata for SR-IOV external ports, you cannot remove or repurpose those nodes.

About this task
Copy link

You can consolidate OVN gateway services onto the RHOCP infrastructure after you migrate to Red Hat OpenStack Services on OpenShift (RHOSO) 18.0.

Procedure
Copy link

  1. Verify that no nicMappings are configured and that no OVN pods are currently running: 
    $ oc get -n openstack $(oc get oscp -o name) \
-o json| jq .spec.ovn.template.ovnController.nicMappings

    The output should show null.

  2. Verify that no OVN controller pods exist: 
    $ oc get pods -n openstack -l service=ovn-controller-ovs

    The output should show No resources found in openstack namespace.

  3. Check if availability zones are configured for your network agents: 
    $ oc exec -n openstack openstackclient -- openstack network agent list\
 --agent-type ovn-controller-gateway -c Host -c "Availability Zone"

    Make a note of any availability zones in the output. You will need this information in a later step.

  4. Configure the OVN Controller in the OpenStackControlPlane CR with the appropriate NIC mappings. Choose one of the following options based on your network configuration:
    • For a single NIC configuration without availability zones: 
      $ oc patch -n openstack $(oc get oscp -n openstack -o name) --type=merge --patch '
spec:
  ovn:
    template:
      ovnController:
        nicMappings:
          _<physical_network>_: _<dedicated_NIC>_
'

      where:

      _<physical_network>_
      Replace with the physical network name that you used when creating the provider network with the --provider-physical-network option in RHOSP 17.1.
      _<dedicated_NIC>_
      Replace with the name of the dedicated NIC on RHOCP worker nodes. This NIC must not be shared with any RHOCP or RHOSP networks. The NIC is moved to the OVS pod's network namespace by using the Multus host-device plugin.
    • For a single NIC configuration with availability zones: 
      $ oc patch -n openstack $(oc get oscp -n openstack -o name) --type=merge --patch '
spec:
  ovn:
    template:
      ovnController:
        nicMappings:
          _<physical_network>_: _<dedicated_NIC>_
        external-ids:
          availability-zones: [_<az_1>_,_<az_2>_,_<az_n>_]
'

      where:

      _<az_1>_,_<az_2>_,_<az_n>_
      Specifies the availability zone names from the previous step.
    • For a bonded NIC configuration: 
      $ oc patch -n openstack $(oc get oscp -n openstack -o name) --type=merge --patch '
spec:
  ovn:
    template:
      ovnController:
        bondConfiguration:
          bond0:
            mode: active-backup
            mtu: 1500
            links:
              - _<dedicated_NIC_1>_
              - _<dedicated_NIC_2>_
        nicMappings:
          _<physical_network>_: bond0
'
      where:
      _<dedicated_NIC_1>_ and _<dedicated_NIC_2>_
      Specifies the names of the NICs on RHOCP worker nodes that are dedicated for the RHOSO OVN gateways.
      mode and mtu
      Specifies the mode and MTUs that are needed for your environment.
  5. Verify that the OVS pods are created and running: 
    $ oc get pods -n openstack -l service=ovn-controller-ovs
    Example output 
    NAME                       READY   STATUS    RESTARTS   AGE
ovn-controller-ovs-6v5x4   2/2     Running   0          2m59s
ovn-controller-ovs-rlbr6   2/2     Running   0          112s
ovn-controller-ovs-vnhnw   2/2     Running   0          45s
  6. Confirm that the new OVN Controller Gateway agents are detected: 
    $ oc exec openstackclient -- openstack network agent list --agent-type ovn-controller-gateway
    Example output 
    +--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------+
| ID                                   | Agent Type                   | Host                     | Availability Zone | Alive | State | Binary         |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------+
| bb01f541-4be3-4c01-b83f-4367f771c9e5 | OVN Controller Gateway agent | controller-2.localdomain |                   | :-)   | UP    | ovn-controller |
| 448e20cd-5182-4d60-a9e6-82720bd7202e | OVN Controller Gateway agent | master-2                 |                   | :-)   | UP    | ovn-controller |
| 6e6dd9fa-8490-4559-b262-8dbc54c6e84a | OVN Controller Gateway agent | master-0                 |                   | :-)   | UP    | ovn-controller |
| 207ccdb1-4264-49a6-ae98-c81e2a821512 | OVN Controller Gateway agent | master-1                 |                   | :-)   | UP    | ovn-controller |
| 0e53e998-bcbf-4d0d-8259-0e3aca68c810 | OVN Controller Gateway agent | controller-0.localdomain |                   | :-)   | UP    | ovn-controller |
| 5e0e984a-90ca-4cdb-9056-1246076af62d | OVN Controller Gateway agent | controller-1.localdomain |                   | :-)   | UP    | ovn-controller |
+--------------------------------------+------------------------------+--------------------------+-------------------+-------+-------+----------------+

    The output shows both the old Controller nodes and the new RHOCP worker nodes (for example, master-0, master-1, master-2) running OVN Controller Gateway agents.

  7. Create a custom OpenStackDataPlaneService CR to stop the OVN controller on the old Controller or Networker nodes: 
    $ cat > remove-gateway-chassis-service.yaml <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneService
metadata:
  labels:
    osdps: remove-gateway-chassis
  name: remove-gateway-chassis
  namespace: openstack
spec:
  playbookContents: |
    - hosts: all
      strategy: linear
      become: true
      tasks:
        - name: Remove ovn-cms-options
          ansible.builtin.command: ovs-vsctl remove open . external_ids ovn-cms-options
        - name: Stop and disable ovn_controller
          ansible.builtin.systemd_service:
            name: edpm_ovn_controller
            state: stopped
            enabled: false
        - name: Stop and disable ovn_metadata_agent
          failed_when: false
          ansible.builtin.systemd_service:
            name: edpm_ovn_metadata_agent
            state: stopped
            enabled: false
EOF
  8. Apply the service CR: 
    $ oc apply -f remove-gateway-chassis-service.yaml
  9. Create an OpenStackDataPlaneDeployment CR to run the service on your Networker node set: 
    $ cat > remove-gateway-deployment.yaml <<EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: remove-gateway
  namespace: openstack
spec:
  backoffLimit: 3
  deploymentRequeueTime: 15
  servicesOverride:
    - remove-gateway-chassis
  nodeSets:
  - _<networker_nodeset>_
  preserveJobs: true
EOF
    where:
    _<networker_nodeset>_
    Specifies the name of your Networker node set, for example, openstack-networker.
  10. Apply the deployment CR: 
    $ oc apply -f remove-gateway-deployment.yaml
  11. Wait for the deployment to reach Setup complete status: 
    $ oc get osdpd remove-gateway
    Example output 
    NAME             NODESETS                  STATUS   MESSAGE
remove-gateway   ["openstack-networker"]   True     Setup complete
  12. Delete the stale network agents from all Controller or Networker nodes. Run the following command for each node: 
    $ oc exec -n openstack openstackclient -- openstack network agent \
delete $(oc exec openstackclient -- openstack network agent list \
--host _<hostname>_ -f value -c ID)

    where:

    _<hostname>_
    Specifies the hostname of each Controller or Networker node, for example, controller-0.localdomain, controller-1.localdomain, controller-2.localdomain.
    Example 
    $ oc exec openstackclient -- openstack network agent delete $(oc exec openstackclient -- openstack network agent list --host controller-0.localdomain -f value -c ID)
$ oc exec openstackclient -- openstack network agent delete $(oc exec openstackclient -- openstack network agent list --host controller-1.localdomain -f value -c ID)
$ oc exec openstackclient -- openstack network agent delete $(oc exec openstackclient -- openstack network agent list --host controller-2.localdomain -f value -c ID)

Results
Copy link

  1. Verify that all router gateway ports are now hosted on the new gateway chassis nodes running on RHOCP worker nodes: 
    $ oc exec -n openstack ovsdbserver-sb-0 -- ovn-sbctl --no-leader-only \
show | grep -e hostname -e cr-lrp
    Example output 
        hostname: master-2
    Port_Binding cr-lrp-cd37a0e1-c6be-45bc-8a4d-7e58f7ae86e5
    Port_Binding cr-lrp-e486080d-06af-45c8-9046-0cbea16d8007
    Port_Binding cr-lrp-53a77212-d0d1-4bb1-a093-9ce81646f7e1
    hostname: compute-1.localdomain
    hostname: master-0
    Port_Binding cr-lrp-2ee38759-832f-4ca9-976f-d9a63c412b69
    Port_Binding cr-lrp-8dcd5a5b-30ac-43a8-b912-b0094349e5c0
    Port_Binding cr-lrp-5f87a35e-daea-45f2-be3b-89564f9e4a8c
    hostname: compute-0.localdomain
    hostname: master-1
    Port_Binding cr-lrp-083dd00f-c923-45b1-9942-07e29c58e762
    Port_Binding cr-lrp-21a031f7-5352-44d9-bf70-da2e5fc76a1b
    Port_Binding cr-lrp-5c2f5834-0f89-4a6c-bbfd-ca20572aad47

    In this example, all router gateway ports (cr-lrp-*) are now bound to the RHOCP worker nodes (master-0, master-1, master-2) instead of the old Controller nodes.

  2. Verify that the network agent list shows only the new gateway agents on RHOCP worker nodes: 
    $ oc exec -n openstack openstackclient -- openstack network agent list \
--agent-type ovn-controller-gateway

    Confirm that only the RHOCP worker nodes appear in the output, and that the old Controller or Networker nodes are no longer listed.

  3. If the old Controller or Networker nodes are not being used for any other purpose, you can now release or repurpose that hardware. Note
    When you run the remove-gateway data plane deployment, the router gateway ports are moved to new RHOCP worker nodes. Expect a brief outage of centralized data plane traffic during this migration. A similar outage occurs during RHOCP and RHOSO updates when the OVS pods restart and router gateway ports are moved between nodes.

Enable the high availability for Compute instances service
Copy link

Enable the high availability for Compute instances (Instance HA) service.

Before you begin
Copy link

About this task
Copy link

Create the following resources:

  • Fencing secret.
  • Config map. You can create the config map manually, or the config map is created automatically when you deploy the Instance HA resource. However, you must create the config map manually if you want to disable the Instance HA service.
  • Instance HA resource.

Procedure
Copy link

  1. Create the secret: 
    $ oc apply -f fencing-secret.yaml -n openstack
  2. Optional: Create the Instance HA config map and set the DISABLED parameter to false. For example: 
    $ cat << EOF > iha-cm.yaml
kind: ConfigMap
metadata:
  name: instanceha-0-config
  namespace: openstack
apiVersion: v1
data:
  config.yaml: |
    config:
      EVACUABLE_TAG: "evacuable"
      TAGGED_IMAGES: "true"
      TAGGED_FLAVORS: "true"
      TAGGED_AGGREGATES: "true"
      SMART_EVACUATION: "false"
      DELTA: "30"
      DELAY: "0"
      POLL: "45"
      THRESHOLD: "50"
      WORKERS: "4"
      RESERVED_HOSTS: "false"
      LEAVE_DISABLED: "false"
      CHECK_KDUMP: "false"
      LOGLEVEL: "info"
      DISABLED: "false"
EOF
    1. Apply the configuration: 
      $ oc apply -f iha-cm.yaml -n openstack
      Note

      If you want to restrict which Compute nodes are evacuated, create host aggregates and set them by using the EVACUABLE_TAG parameter. Alternatively, you can set the TAGGED_AGGREGATES parameter to false to enable monitoring and evacuation of all your Compute nodes. For more information about Instance HA service parameters, see Instance HA service parameters.
  3. Create an Instance HA resource and reference the fencing secret and config map. For example: 
    $ cat << EOF > iha.yaml
apiVersion: instanceha.openstack.org/v1beta1
kind: InstanceHa
metadata:
  name: instanceha-0
  namespace: openstack
spec:
  caBundleSecretName: combined-ca-bundle
  instanceHaConfigMap:
  fencingSecret: fencing-secret
EOF
    • spec.instanceHaConfigMap defines the name of the YAML file containing the Instance HA config map that you created. If you do not create this file, then a YAML file called instanceha-config is created automatically when the Instance HA service is installed, providing the default values of the Instance HA service parameters. You can then edit the values as needed.
  4. Deploy the Instance HA resource: 
    $ oc apply -f iha.yaml -n openstack

What to do next
Copy link

  • After you complete the Red Hat OpenStack Services on OpenShift adoption, remove the Pacemaker components from the Compute nodes. You must run the following commands on each Compute node: 
    $ sudo systemctl stop pacemaker_remote
$ sudo systemctl stop pcsd
$ sudo systemctl stop pcsd-ruby.service
$ sudo systemctl disable pacemaker_remote
$ sudo systemctl disable pcsd
$ sudo systemctl disable pcsd-ruby.service
$ sudo dnf remove pacemaker pacemaker-remote pcs pcsd -y