Back up the control plane

To prepare your RHOSO deployment for disaster recovery, you create OADP backups of persistent volumes and Kubernetes resources such as the databases, persistent data, and configuration resources that are required to restore your environment.

Note
  • Global resources and resources created in non-RHOSO namespaces, such as NodeNetworkConfigPolicy (nncp) CRs, are not included in the resources that are backed up.
  • You cannot perform a backup while performing a minor update.

Prerequisites
Copy link

  • You are logged in to the RHOCP cluster as a user with cluster-admin privileges.
  • You have configured a GaleraBackup CR for each Galera database instance.

Collect backup metadata
Copy link

To verify Operator version compatibility when restoring your environment from a backup, you must annotate the backup resources with metadata, including a unique backup timestamp.

About this task
Copy link

Note
Ensure that any minor updates are finished successfully before beginning a backup.

Procedure
Copy link

  1. Verify that the RHOSO control plane is in the "Ready" state: 
    $ oc wait openstackcontrolplane -n openstack --all \
 --for=condition=Ready --timeout=10s
  2. Verify that the RHOSO data plane is in the "Ready" state: 
    $ oc wait openstackdataplanenodeset -n openstack --all \
 --for=condition=Ready --timeout=10s
  3. Count the labeled resources to ensure that the OpenStackBackupConfig resource is labeling resources correctly: 
    $ echo "Secrets: $(oc get secret -n openstack -l backup.openstack.org/restore=true --no-headers | wc -l)"
$ echo "ConfigMaps: $(oc get configmap -n openstack -l backup.openstack.org/restore=true --no-headers | wc -l)"
$ echo "PVCs: $(oc get pvc -n openstack -l backup.openstack.org/backup=true --no-headers | wc -l)"

    Verify that each command returns a count greater than zero. If any count is zero, the OpenStackBackupConfig might not be functioning correctly.

  4. Create and record a timestamp for the backup: 
    $ export BACKUP_TS=$(date +%Y%m%d-%H%M%S)

    Record this timestamp value for use throughout the backup procedure.

  5. Retrieve and record the OpenStack Operator version: 
    $ export CSV_VERSION=$(oc get csv -n openstack-operators \
  -l operators.coreos.com/openstack-operator.openstack-operators \
  -o jsonpath='{.items[0].metadata.name}')
  6. Retrieve and record the catalog source image: 
    $ export CATALOG_IMG=$(oc get catalogsource -n openshift-marketplace \
 redhat-operators -o jsonpath='{.spec.image}')
  7. Retrieve and record the Operator image: 
    $ export OPERATOR_IMG=$(oc get deployment openstack-operator-controller-manager \
  -n openstack-operators -o jsonpath='{.spec.template.spec.containers[0].image}')
  8. Verify that the variables have recorded valid metadata values: 
    $ echo "Backup timestamp: ${BACKUP_TS}"
$ echo "CSV version: ${CSV_VERSION}"
$ echo "Catalog image: ${CATALOG_IMG}"
$ echo "Operator image: ${OPERATOR_IMG}"

What to do next
Copy link

  • Backup the Galera databases.

Back up the Galera databases
Copy link

To generate SQL dump files in persistent volumes for your OADP PVC backup, create a manual job from each GaleraBackup cronjob.

Procedure
Copy link

  1. Trigger backup jobs for all Galera database instances and inject the backup timestamp as an environment variable: 
    $ for CRONJOB in $(oc get cronjob -n openstack -l app=galera \
  -o jsonpath='{.items[*].metadata.name}'); do
  JOB_NAME="${CRONJOB}-${BACKUP_TS}"
  oc -n openstack create job --from=cronjob/${CRONJOB} ${JOB_NAME} \
    --dry-run=client -o json | \
    jq '.spec.template.spec.containers[0].env += [{"name":"BACKUP_TIMESTAMP","value":"'${BACKUP_TS}'"}]' | \
    oc -n openstack create -f -
done
  2. Wait for the Galera backup jobs to complete: 
    $ oc wait --for=condition=complete job -l app=galera --timeout=10m -n openstack

    All jobs should show 1/1 completions.

  3. Optional: Verify that backup files were created in the Galera persistent volumes: 
    $ oc get galerabackup
NAME   READY   MESSAGE
foo    True    Setup complete

$ oc get jobs |grep foo
backup-foo-20260610-100600    Complete   1/1     20s     12m

$ oc debug  jobs/backup-foo-20260610-100600 -- ls -la /backup/data/
-rw-rw-r--. 1 mysql mysql 252696 Jun 10 10:06 openstack_backup_20260610-100600.sql.gz
-rw-rw-r--. 1 mysql mysql    837 Jun 10 10:06 openstack_backup-grants_20260610-100600.sql.gz

What to do next
Copy link

  • Optional: Backup your OVN database.
  • Create the OADP PVC backup.

Back up OVN databases
Copy link

To reduce restoration time by avoiding manual reconstruction from the Networking service (neutron) data, back up your Open Virtual Network (OVN) northbound and southbound databases. The backup files are written directly to the OVN PVCs.

About this task
Copy link

If you backup the OVN databases, you must perform the OVN database restore procedure during restoration. If you do not backup the OVN databases, you must synchronise the Networking service with OVN during restoration to reconstruct the OVN databases.

Note
If you do not backup the OVN databases, virtual machine instance network connectivity might be interrupted during restoration as network services restart.

Procedure
Copy link

  1. Verify that the OVN database servers are running and healthy: 
    $ oc wait ovndbclusters -n openstack --all --for condition=Ready
  2. Label the OVN northbound database PVCs for backup: 
    $ oc label pvc -n openstack -l service=ovsdbserver-nb \
 backup.openstack.org/backup=true \
 backup.openstack.org/restore=true \
 backup.openstack.org/restore-order=00 \
 backup.openstack.org/category=controlplane
  3. Label the OVN southbound database PVCs for backup: 
    $ oc label pvc -n openstack -l service=ovsdbserver-sb \
 backup.openstack.org/backup=true \
 backup.openstack.org/restore=true \
 backup.openstack.org/restore-order=00 \
 backup.openstack.org/category=controlplane
  4. Create a backup of the northbound database: 
    $ oc exec ovsdbserver-nb-0 -n openstack -- bash -c \
 "ovsdb-client backup unix:/etc/ovn/ovnnb_db.sock OVN_Northbound \
   > /etc/ovn/ovnnb_db.db.${BACKUP_TS}"
  5. Create a backup of the southbound database: 
    $ oc exec ovsdbserver-sb-0 -n openstack -- bash -c \
 "ovsdb-client backup unix:/etc/ovn/ovnsb_db.sock OVN_Southbound \
   > /etc/ovn/ovnsb_db.db.${BACKUP_TS}"
  6. Verify that the backup files were created: 
    $ oc exec ovsdbserver-nb-0 -n openstack -- ls -lh /etc/ovn/ovnnb_db.db.${BACKUP_TS}
$ oc exec ovsdbserver-sb-0 -n openstack -- ls -lh /etc/ovn/ovnsb_db.db.${BACKUP_TS}

    Both commands should display file size information for the backup files.

What to do next
Copy link

  • Create the OADP PVC backup.

Create an OADP PVC backup
Copy link

To preserve your database exports, Image service (glance) images, and other persistent data, capture all persistent volume claims (PVCs) that are labeled for backup in your environment.

Procedure
Copy link

  1. Create a Backup custom resource (CR) file for the PVC backup named backup-pvcs.yaml: 
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: openstack-backup-pvcs-${BACKUP_TS}
  namespace: openshift-adp
  annotations:
    openstack.org/csv-version: "${CSV_VERSION}"
    openstack.org/catalog-source-image: "${CATALOG_IMG}"
    openstack.org/operator-image: "${OPERATOR_IMG}"
spec:
  includedNamespaces:
  - openstack
  labelSelector:
    matchLabels:
      backup.openstack.org/backup: "true"
  snapshotVolumes: true
  defaultVolumesToFsBackup: false
  snapshotMoveData: true
  volumeSnapshotLocations: []
  storageLocation: velero-1
  ttl: 720h
  hooks:
    resources:
      - name: swift-xattr-backup
        labelSelector:
          matchLabels:
            component: swift-storage
        pre:
          - exec:
              container: object-server
              command:
                - /bin/bash
                - -c
                - |
                  set -e
                  DUMP="/srv/node/pv/.swift-xattrs.dump"
                  rm -f "$DUMP" "${DUMP}.applied" "${DUMP}.missing"
                  getfattr -R -d -m user.swift /srv/node/pv/ 1> "$DUMP"
                  echo "xattr backup complete: $(grep -c '^# file:' "$DUMP") files"
              onError: Fail
              timeout: 600s
    • spec.labelSelector.matchLabels.backup.openstack.org/backup: Set to "true" to match only PVCs labeled by Operators.
    • spec.snapshotVolumes: Set to true to use CSI volume snapshots.
    • spec.defaultVolumesToFsBackup: Set to false to disable file system backup.
    • spec.snapshotMoveData: Set to true to upload snapshots to S3 by using Data Mover. Remove this field if you do not want to upload snapshot data to the BackupStorageLocation (local CSI snapshots only). Without Data Mover, PVC restore requires the original VolumeSnapshotContent to still exist.
    • spec.volumeSnapshotLocations: Specifies the volume snapshot location. Leave empty to use the default CSI driver. Depending on the storage backend, you might need to configure volumeSnapshotLocations or adjust CSI driver settings.
    • spec.storageLocation: Specifies the name of the BackupStorageLocation CR.
    • spec.ttl: Specifies how long to retain the backup in hours, for example, set to 720h to retain backups for 30 days.
    • hooks.resources: Required only when the Object Storage service (swift) is deployed as part of the control plane. Specifies how to export swift extended attributes (xattrs) to a file on the PVC before the CSI snapshot. The Object Storage service stores all object metadata in xattrs. OADP Data Mover uses kopia for filesystem-level data transfer, which does not preserve xattrs. The hook is safe to include even if the Object Storage service is not deployed. The hook only targets pods with component: swift-storage and is silently skipped if no matching pods exist.
  2. Create the Backup CR, substituting environment variables: 
    $ envsubst '${BACKUP_TS} ${CSV_VERSION} ${CATALOG_IMG} ${OPERATOR_IMG}' < backup-pvcs.yaml | oc apply -f -
  3. Wait for the backup to complete: 
    $ oc wait --for=jsonpath='{.status.phase}'=Completed \
  backup/openstack-backup-pvcs-${BACKUP_TS} -n openshift-adp --timeout=30m

    The backup might take 30 minutes or longer depending on the size of your persistent volumes.

  4. Verify that the backup completed successfully: 
    $ oc get backup openstack-backup-pvcs-${BACKUP_TS} -n openshift-adp -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

    The phase should be Completed.

  5. Verify that Data Mover uploads completed: 
    $ oc get datauploads -n openshift-adp \
  -l velero.io/backup-name=openstack-backup-pvcs-${BACKUP_TS} \
  -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

    All data uploads should show Completed phase.

What to do next
Copy link

  • Create the OADP resources backup.

Create an OADP resources backup
Copy link

To preserve your Secrets, ConfigMaps, custom resources, and other configuration objects, capture all Kubernetes resources except persistent volume claims.

Procedure
Copy link

  1. Create a Backup custom resource (CR) file for the resources backup named backup-resources.yaml: 
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: openstack-backup-resources-${BACKUP_TS}
  namespace: openshift-adp
  annotations:
    openstack.org/csv-version: "${CSV_VERSION}"
    openstack.org/catalog-source-image: "${CATALOG_IMG}"
    openstack.org/operator-image: "${OPERATOR_IMG}"
spec:
  includedNamespaces:
  - openstack
  excludedResources:
  - persistentvolumeclaims
  - persistentvolumes
  storageLocation: velero-1
  ttl: 720h
  2. Create the Backup CR, substituting environment variables: 
    $ envsubst '${BACKUP_TS} ${CSV_VERSION} ${CATALOG_IMG} ${OPERATOR_IMG}' < backup-pvcs.yaml | oc apply -f -
  3. Wait for the backup to complete: 
    $ oc wait --for=jsonpath='{.status.phase}'=Completed \
  backup/openstack-backup-resources-${BACKUP_TS} -n openshift-adp --timeout=30m
  4. Verify that the backup completed successfully: 
    $ oc get backup openstack-backup-resources-${BACKUP_TS} -n openshift-adp -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

    The phase should be Completed.

  5. Verify that the backup includes the version annotations: 
    $ oc get backup openstack-backup-resources-${BACKUP_TS} -n openshift-adp \
 -o custom-columns=\
'NAME:.metadata.name,'\
'CSV:.metadata.annotations.openstack\.org/csv-version,'\
'CATALOG:.metadata.annotations.openstack\.org/catalog-source-image'

    All three columns should display values.

What to do next
Copy link

  • Optional: If you are using BaremetalHost resources that are not in the openstack namespace and your OpenStackDataPlaneNodeSet CRs do not use preProvisioned: true, create a backup of your BaremetalHost resources.
  • Verify all backups were successful.

Create an OADP backup for bare-metal data plane nodes
Copy link

If your OpenStackDataPlaneNodeSet custom resources (CRs) use unprovisioned, bare-metal nodes, then you must create a backup of the BaremetalHost resources if they are located in a namespace other than openstack.

Before you begin
Copy link

  • You are not using preProvisioned: true in your OpenStackDataPlaneNodeSet CRs.

About this task
Copy link

Note
The OpenStackBaremetalSet and OpenStackProvisionServer resources are in the openstack namespace and are already captured by the OADP resources backup.

Procedure
Copy link

  1. Set the namespace where BaremetalHost resources are located: 
    $ export BMH_NAMESPACE=<namespace>
$ export BMH_BACKUP=openstack-backup-bmh-${BACKUP_TS}

    where:

    <namespace>
    Specifies the name of the namespace that contains your BaremetalHost resources.
  2. Create a Backup CR file for the BaremetalHost resources backup named backup-baremetal.yaml: 
    apiVersion: velero.io/v1
kind: Backup
metadata:
  name: ${BMH_BACKUP}
  namespace: openshift-adp
  annotations:
    openstack.org/csv-version: "${CSV_VERSION}"
    openstack.org/catalog-source-image: "${CATALOG_IMG}"
    openstack.org/operator-image: "${OPERATOR_IMG}"
spec:
  includedNamespaces:
  - ${BMH_NAMESPACE}
  includedResources:
  - baremetalhosts.metal3.io
  - secrets
  - configmaps
  storageLocation: velero-1
  ttl: 720h
  3. Create the Backup CR, substituting environment variables: 
    $ envsubst '${BMH_BACKUP} ${BMH_NAMESPACE} ${CSV_VERSION} ${CATALOG_IMG} ${OPERATOR_IMG}' < backup-baremetal.yaml | oc apply -f -
  4. Wait for the backup to complete: 
    $ oc wait --for=jsonpath='{.status.phase}'=Completed \
  backup/${BMH_BACKUP} -n openshift-adp --timeout=30m
  5. Verify that the backup completed successfully: 
    $ oc get backup ${BMH_BACKUP} -n openshift-adp -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

    The phase should be Completed.

What to do next
Copy link

  • Verify all backups were successful.

Verify successful backup completion
Copy link

To ensure your environment is ready for future restore operations, confirm that all backup data successfully uploaded to your object storage.

Procedure
Copy link

  1. Verify the status of all backups: 
    $ oc get backup -n openshift-adp -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

    All backups should show Completed phase.

  2. Verify that backups include the version annotations: 
    $ oc get backup -n openshift-adp -o custom-columns=\
'NAME:.metadata.name,'\
'CSV:.metadata.annotations.openstack\.org/csv-version,'\
'CATALOG:.metadata.annotations.openstack\.org/catalog-source-image'

    All backups should display version information in the CSV and CATALOG columns.

  3. Verify that Data Mover uploads completed for the PVC backup: 
    $ oc get datauploads -n openshift-adp \
 -l velero.io/backup-name=openstack-backup-pvcs-${BACKUP_TS} \
 -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

    All data uploads should show Completed phase.

  4. Record the backup timestamp (BACKUP_TS value) in a safe location for future reference during restore operations.
  5. Optional: Create additional backups on a regular schedule using cron jobs or other automation.