$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
You can debug Velero custom resources (CRs) by using the OpenShift CLI tool or the Velero CLI tool. The Velero CLI tool provides more detailed logs and information.
You can check installation issues, backup and restore CR issues, and Restic issues.
You can collect logs, CR information, and Prometheus metric data by using the must-gather
tool.
You can obtain the Velero CLI tool by:
Downloading the Velero CLI tool
Accessing the Velero binary in the Velero deployment in the cluster
You can download and install the Velero CLI tool by following the instructions on the Velero documentation page.
The page includes instructions for:
macOS by using Homebrew
GitHub
Windows by using Chocolatey
You have access to a Kubernetes cluster, v1.16 or later, with DNS and container networking enabled.
You have installed kubectl
locally.
Open a browser and navigate to "Install the CLI" on the Verleo website.
Follow the appropriate procedure for macOS, GitHub, or Windows.
Download the Velero version appropriate for your version of OADP, according to the table that follows:
OADP version | Velero version |
---|---|
0.2.6 |
1.6.0 |
0.5.5 |
1.7.1 |
1.0.0 |
1.7.1 |
1.0.1 |
1.7.1 |
1.0.2 |
1.7.1 |
1.0.3 |
1.7.1 |
You can use a shell command to access the Velero binary in the Velero deployment in the cluster.
Your DataProtectionApplication
custom resource has a status of Reconcile complete
.
Enter the following command to set the needed alias:
$ alias velero='oc -n openshift-adp exec deployment/velero -c velero -it -- ./velero'
You can debug a failed backup or restore by checking Velero custom resources (CRs) and the Velero
pod log with the OpenShift CLI tool.
Use the oc describe
command to retrieve a summary of warnings and errors associated with a Backup
or Restore
CR:
$ oc describe <velero_cr> <cr_name>
Use the oc logs
command to retrieve the Velero
pod logs:
$ oc logs pod/<velero>
You can specify the Velero log level in the DataProtectionApplication
resource as shown in the following example.
This option is available starting from OADP 1.0.3. |
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: velero-sample
spec:
configuration:
velero:
logLevel: warning
The following logLevel
values are available:
trace
debug
info
warning
error
fatal
panic
It is recommended to use debug
for most logs.
You can debug Backup
and Restore
custom resources (CRs) and retrieve logs with the Velero CLI tool.
The Velero CLI tool provides more detailed information than the OpenShift CLI tool.
Use the oc exec
command to run a Velero CLI command:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
<backup_restore_cr> <command> <cr_name>
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
Use the velero --help
option to list all Velero CLI commands:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
--help
Use the velero describe
command to retrieve a summary of warnings and errors associated with a Backup
or Restore
CR:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
<backup_restore_cr> describe <cr_name>
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
Use the velero logs
command to retrieve the logs of a Backup
or Restore
CR:
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
<backup_restore_cr> logs <cr_name>
$ oc -n openshift-adp exec deployment/velero -c velero -- ./velero \
restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
Velero has limited abilities to resolve admission webhook issues during a restore. If you have workloads with admission webhooks, you might need to use an additional Velero plugin or make changes to how you restore the workload.
Typically, workloads with admission webhooks require you to create a resource of a specific kind first. This is especially true if your workload has child resources because admission webhooks typically block child resources.
For example, creating or restoring a top-level object such as service.serving.knative.dev
typically creates child resources automatically. If you do this first, you will not need to use Velero to create and restore these resources. This avoids the problem of child resources being blocked by an admission webhook that Velero might use.
This section describes the additional steps required to restore resources for several types of Velero backups that use admission webhooks.
You might encounter problems using Velero to back up Knative resources that use admission webhooks.
You can avoid such problems by restoring the top level Service
resource first whenever you back up and restore Knative resources that use admission webhooks.
Restore the top level service.serving.knavtive.dev Service
resource:
$ velero restore <restore_name> \
--from-backup=<backup_name> --include-resources \
service.serving.knavtive.dev
If you experience issues when you use Velero to a restore an IBM AppConnect resource that has an admission webhook, you can run the checks in this procedure.
Check if you have any mutating admission plugins of kind: MutatingWebhookConfiguration
in the cluster:
$ oc get mutatingwebhookconfigurations
Examine the YAML file of each kind: MutatingWebhookConfiguration
to ensure that none of its rules block creation of the objects that are experiencing issues. For more information, see the official Kuberbetes documentation.
Check that any spec.version
in type: Configuration.appconnect.ibm.com/v1beta1
used at backup time is supported by the installed Operator.
You might encounter issues caused by using invalid directories or incorrect credentials when you install the Data Protection Application.
The Velero
pod log displays the error message, Backup storage contains invalid top-level directories
.
The object storage contains top-level directories that are not Velero directories.
If the object storage is not dedicated to Velero, you must specify a prefix for the bucket by setting the spec.backupLocations.velero.objectStorage.prefix
parameter in the DataProtectionApplication
manifest.
The oadp-aws-registry
pod log displays the error message, InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
The Velero
pod log displays the error message, NoCredentialProviders: no valid providers in chain
.
The credentials-velero
file used to create the Secret
object is incorrectly formatted.
Ensure that the credentials-velero
file is correctly formatted, as in the following example:
credentials-velero
file[default] (1) aws_access_key_id=AKIAIOSFODNN7EXAMPLE (2) aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
1 | AWS default profile. |
2 | Do not enclose the values with quotation marks (" , ' ). |
You might encounter these common issues with Backup
and Restore
custom resources (CRs).
The Backup
CR displays the error message, InvalidVolume.NotFound: The volume ‘vol-xxxx’ does not exist
.
The persistent volume (PV) and the snapshot locations are in different regions.
Edit the value of the spec.snapshotLocations.velero.config.region
key in the DataProtectionApplication
manifest so that the snapshot location is in the same region as the PV.
Create a new Backup
CR.
The status of a Backup
CR remains in the InProgress
phase and does not complete.
If a backup is interrupted, it cannot be resumed.
Retrieve the details of the Backup
CR:
$ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
backup describe <backup>
Delete the Backup
CR: