Troubleshoot volume backups
You can diagnose many issues by verifying that the Block Storage services are running correctly and then by examining the log files for error messages.
Backup troubleshooting tips Copy linkLink copied!
The Block Storage backup service performs static checks on backup requests and notifies you immediately when a check fails.
The service checks for the following conditions:
- Check for an invalid volume reference (
missing). - Check if the volume is
in-useor attached to an instance. Thein-usecase requires you to use the--forceoption to perform a backup.When you use the
--forcevolume backup option, you create a crash-consistent, but not an application-consistent, backup because the volume is not quiesced before performing the backup. Therefore, the data is intact but the backup does not have an awareness of which applications were running when the backup was performed.
When these checks succeed: the Block Storage backup service accepts the request to backup this volume, the CLI backup command returns immediately, and the volume is backed up in the background.
Therefore the CLI backup command returns even if the backup fails. You can use the openstack volume backup list command to verify that the volume backup is successful, when the Status of the backup entry is available.
If a backup fails, examine the Block Storage backup service log file for error messages to discover the cause.
Examine the backup service log Copy linkLink copied!
When a backup or restore does not succeed, you can examine the Block Storage backup service log file for error messages that can help you to determine the reason.
Procedure Copy linkLink copied!
This log file is located in the following path: /var/log/containers/cinder/cinder-backup.log.
Volume backup workflow Copy linkLink copied!
Review the volume backup workflow when users request backups through the Block Storage service (cinder) API.
- The user issues a request to the cinder API, which is a REST API, to back up a Block Storage volume.
- The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information.
- The cinder API creates the backup record in the SQL database.
- The cinder API makes an RPC call to the
cinder-scheduler. - The
cinder-schedulermakes an asynchronous RPC call to thecinder-backupservice via AMQP to back up the volume. - The cinder API returns the current backup record, with an ID, to the API caller.
- An RPC create message arrives on one of the backup services.
- The
cinder-backupservice performs a synchronous RPC call toget_backup_device. - The
cinder-volumeservice ensures that the correct device is returned to the caller. Normally, it is the same volume, but if the volume is in use, the service returns a temporary cloned volume or a temporary snapshot, depending on the configuration. - The
cinder-backupservice issues another synchronous RPC tocinder-volumeto expose the source device. - The
cinder-volumeservice exports and maps the source device (volume or snapshot) and returns the appropriate connection information. - The
cinder-backupservice attaches the source device by using the connection information. - The
cinder-backupservice calls the backup back end driver, with the device already attached, which begins the data transfer to the backup repository. - The source device is detached from the Backup host.
- The
cinder-backupservice issues a synchronous RPC tocinder-volumeto disconnect the source device. - The
cinder-volumeservice unmaps and removes the export for the device. - If a temporary volume or temporary snapshot was created,
cinder-backupcallscinder-volumeto remove it. - The
cinder-volumeservice removes the temporary volume. - When the backup is completed, the backup record is updated in the database.
Volume restore workflow Copy linkLink copied!
Review the volume restore workflow when users request backup restoration through the Block Storage service (cinder) API.
- The user issues a request to the cinder API, which is a REST API, to restore a Block Storage backup.
- The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information.
- If the request does not contain an existing volume as the destination, the cinder API makes an asynchronous RPC call to create a new volume and polls the status of the volume until it becomes available.
- The
cinder-schedulerselects a volume service and makes the RPC call to create the volume. - The selected
cinder-volumeservice creates the volume. - When the cinder API detects that the volume is available, the backup record is updated in the database.
- The cinder API makes an asynchronous RPC call to the backup service via AMQP to restore the backup.
- The cinder API returns the current volume ID, backup ID, and volume name to the API caller.
- An RPC create message arrives on one of the backup services.
- The
cinder-backupservice performs a synchronous RPC call tocinder-volumeto expose the volume. - The
cinder-volumeservice exports and maps the volume returning the appropriate connection information. - The
cinder-backupservice attaches the volume by using the connection information. - The
cinder-backupservice calls the back end driver with the volume already attached, which begins the data restoration to the volume. - The volume is detached from the backup host.
- The
cinder-backupservice issues a synchronous RPC tocinder-volumeto disconnect the volume. - The
cinder-volumeservice unmaps and removes the export for the volume. - When the volume is restored, the backup record is updated in the database.