Rebalance Object Storage rings

When you change the Object Storage service (swift) topology, the swift-operator rebalances the rings. For external data plane nodes, push the updated rings to the data plane nodes. After a rebalance, verify that all data has been moved to its new location before performing another rebalance.

Before you begin
Copy link

  • You have populated the dispersion tool before the rebalance. You only need to run this command once per cluster: 
    $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-dispersion-populate'

    This creates a set of test objects and containers that are distributed across all partitions in the ring. The dispersion tool uses these to verify that all replicas are in the expected locations. You must run this command before a rebalance so that the test objects are placed according to the current ring layout.

About this task
Copy link

Note

Successive rebalances must respect the minPartHours interval. A rebalance is only possible when swift-ring-builder shows 0:00:00 remaining.

Procedure
Copy link

  1. Manually trigger a ring rebalance:
    Important

    Only rebalance after any previous rebalance is fully replicated. Rebalancing before replication has completed can cause data to be moved multiple times, increasing the risk of temporary data unavailability.

     
    $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-ring-tool rebalance'

    This rebalances all Object Storage rings and updates the ring files in the swift-ring-files ConfigMap.

  2. If you use external data plane nodes, push the updated rings to the data plane nodes. Delete any previous ring update deployment and create a new OpenStackDataPlaneDeployment CR that runs only the swift service with the swiftrings Ansible tag. This distributes the updated ring files to all data plane nodes without running a full deployment: 
    $ oc delete --ignore-not-found --wait openstackdataplanedeployment swift-ring-update -n openstack
$ oc apply -n openstack -f - << EOF
apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: swift-ring-update
spec:
  ansibleTags: swiftrings
  servicesOverride:
    - swift
  nodeSets:
    - <nodeset_name>
EOF
    • Replace <nodeset_name> with the name of your OpenStackDataPlaneNodeSet CR.
  3. Wait for the deployment to complete: 
    $ oc wait openstackdataplanedeployment swift-ring-update --for condition=Ready --timeout=300s -n openstack
  4. Verify cluster health by checking that all rings are consistent and replication is progressing: 
    $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-recon -rT --md5'
    • Verify that all ring md5sums are consistent across nodes.
    • Check the replication timestamps. The oldest completion time minus the highest replication time gives you the earliest start time of the last replication pass. This earliest start time must be after the ring ConfigMaps have been updated, confirming that all nodes have completed a full replication pass with the new rings.
  5. Alternatively, you can check the object-replicator logs directly to verify that a full replication pass has completed: 
    $ oc logs --timestamps -l component=swift-storage -c object-replicator --tail 1000 | grep replicated

    Look for log entries that show the completion of a replication pass, for example:

    $ oc logs --timestamps -c object-replicator swift-storage-0
2026-04-02T10:44:13.863401909Z object-replicator: Object replication complete.  (33.00 minutes)

    The completion timestamp minus the duration gives you the start time of the pass. For example, a completion at 10:44:13 with a duration of 33.00 minutes means the pass started at 10:11:13. This start time must be after the ring ConfigMaps were updated. If the pass started before the ConfigMap update, it was still using the old rings and you must wait for the next full pass to complete.

  6. Run swift-dispersion-report to verify that all replicas have been placed correctly: 
    $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-dispersion-report'

    The output shows the percentage of objects and containers with all replicas found. All values must be at 100% before you perform another rebalance.

Custom rings
Copy link

You can create custom rings to update existing Object Storage service (swift) clusters.

When you add new nodes to a cluster, their characteristics might differ from those of the original nodes. Without custom adjustments, the larger capacity of the new nodes might be underused, or if weights change in the rings, data dispersion can become uneven, which reduces safety.

The ring builder helps manage Object Storage as clusters grow and technologies evolve. For assistance with creating custom rings, contact Red Hat Support.