Troubleshoot volume backups

You can diagnose many issues by verifying that the Block Storage services are running correctly and then by examining the log files for error messages.

Backup troubleshooting tips

The Block Storage backup service performs static checks on backup requests and notifies you immediately when a check fails.

The service checks for the following conditions:

  • Check for an invalid volume reference (missing).
  • Check if the volume is in-use or attached to an instance. The in-use case requires you to use the --force option to perform a backup.

    When you use the --force volume backup option, you create a crash-consistent, but not an application-consistent, backup because the volume is not quiesced before performing the backup. Therefore, the data is intact but the backup does not have an awareness of which applications were running when the backup was performed.

When these checks succeed: the Block Storage backup service accepts the request to backup this volume, the CLI backup command returns immediately, and the volume is backed up in the background.

Therefore the CLI backup command returns even if the backup fails. You can use the openstack volume backup list command to verify that the volume backup is successful, when the Status of the backup entry is available.

If a backup fails, examine the Block Storage backup service log file for error messages to discover the cause.

Examine the backup service log

When a backup or restore does not succeed, you can examine the Block Storage backup service log file for error messages that can help you to determine the reason.

Procedure

Find the Block Storage backup service log file on the Controller node where the backup service is running.

This log file is located in the following path: /var/log/containers/cinder/cinder-backup.log.

Volume backup workflow

Review the volume backup workflow when users request backups through the Block Storage service (cinder) API.

Figure 1. Creating a backup of a Block Storage volume

OpenStack BlockStorage backup

  1. The user issues a request to the cinder API, which is a REST API, to back up a Block Storage volume.
  2. The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information.
  3. The cinder API creates the backup record in the SQL database.
  4. The cinder API makes an RPC call to the cinder-scheduler.
  5. The cinder-scheduler makes an asynchronous RPC call to the cinder-backup service via AMQP to back up the volume.
  6. The cinder API returns the current backup record, with an ID, to the API caller.
  7. An RPC create message arrives on one of the backup services.
  8. The cinder-backup service performs a synchronous RPC call to get_backup_device.
  9. The cinder-volume service ensures that the correct device is returned to the caller. Normally, it is the same volume, but if the volume is in use, the service returns a temporary cloned volume or a temporary snapshot, depending on the configuration.
  10. The cinder-backup service issues another synchronous RPC to cinder-volume to expose the source device.
  11. The cinder-volume service exports and maps the source device (volume or snapshot) and returns the appropriate connection information.
  12. The cinder-backup service attaches the source device by using the connection information.
  13. The cinder-backup service calls the backup back end driver, with the device already attached, which begins the data transfer to the backup repository.
  14. The source device is detached from the Backup host.
  15. The cinder-backup service issues a synchronous RPC to cinder-volume to disconnect the source device.
  16. The cinder-volume service unmaps and removes the export for the device.
  17. If a temporary volume or temporary snapshot was created, cinder-backup calls cinder-volume to remove it.
  18. The cinder-volume service removes the temporary volume.
  19. When the backup is completed, the backup record is updated in the database.

Volume restore workflow

Review the volume restore workflow when users request backup restoration through the Block Storage service (cinder) API.

Figure 2. Restoring a Block Storage backup

OpenStack BlockStorage restore

  1. The user issues a request to the cinder API, which is a REST API, to restore a Block Storage backup.
  2. The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information.
  3. If the request does not contain an existing volume as the destination, the cinder API makes an asynchronous RPC call to create a new volume and polls the status of the volume until it becomes available.
  4. The cinder-scheduler selects a volume service and makes the RPC call to create the volume.
  5. The selected cinder-volume service creates the volume.
  6. When the cinder API detects that the volume is available, the backup record is updated in the database.
  7. The cinder API makes an asynchronous RPC call to the backup service via AMQP to restore the backup.
  8. The cinder API returns the current volume ID, backup ID, and volume name to the API caller.
  9. An RPC create message arrives on one of the backup services.
  10. The cinder-backup service performs a synchronous RPC call to cinder-volume to expose the volume.
  11. The cinder-volume service exports and maps the volume returning the appropriate connection information.
  12. The cinder-backup service attaches the volume by using the connection information.
  13. The cinder-backup service calls the back end driver with the volume already attached, which begins the data restoration to the volume.
  14. The volume is detached from the backup host.
  15. The cinder-backup service issues a synchronous RPC to cinder-volume to disconnect the volume.
  16. The cinder-volume service unmaps and removes the export for the volume.
  17. When the volume is restored, the backup record is updated in the database.