Check Object Storage cluster health

The Object Storage service (swift) runs many processes in the background to ensure long-term data availability, durability, and persistence. For example:

About this task

  • Auditors constantly re-read database and object files and compare them by using checksums to make sure there is no silent bit-rot. Any database or object file that no longer matches its checksum is quarantined and becomes unreadable on that node. The replicators then copy one of the other replicas to make the local copy available again.
  • Objects and files can disappear when you replace disks or nodes or when objects are quarantined. When this happens, replicators copy missing objects or database files to one of the other nodes.

The Object Storage service includes a tool called swift-recon that collects data from all nodes and checks the overall cluster health. You can use the swift-recon command line utility to obtain metrics from the account, container, and object servers.

Procedure

  1. Log in to one of the Controller nodes.
  2. Run the following command: 
    $ oc debug --keep-labels=true job/swift-ring-rebalance -- /bin/sh -c 'swift-ring-tool get && swift-recon -arqlT --md5'
    • Optional: Use the --all option to return additional output.

      This command queries all servers on the ring for the following data:

      • Async pendings: If the cluster load is too high and processes cannot update database files fast enough, some updates occur asynchronously. These numbers decrease over time as the load normalizes.
      • Replication metrics: Review the replication timestamps; full replication passes happen frequently with few errors. An old entry, for example, an entry with a timestamp from six months ago, indicates that replication on the node has not completed in the last six months.
      • Ring md5sums: This ensures that all ring files are consistent on all nodes.
      • swift.conf md5sums: This ensures that all configuration files are consistent on all nodes.
      • Quarantined files: There must be no, or very few, quarantined files for all nodes.
      • Time-sync: All nodes must be synchronized.