OADP 1.2 release notes - OADP Application backup and restore | Backup and restore

The release notes for OpenShift API for Data Protection (OADP) 1.2 describe new features and enhancements, deprecated features, product recommendations, known issues, and resolved issues.

OADP 1.2.5 release notes

OpenShift API for Data Protection (OADP) 1.2.5 is a Container Grade Only (CGO) release, released to refresh the health grades of the containers, with no changes to any code in the product itself compared to that of OADP 1.2.4.

Resolved issues

CVE-2023-2431: oadp-velero-plugin-for-microsoft-azure-container: Bypass of seccomp profile enforcement

A flaw was found in Kubernetes, which impacts earlier versions of OADP. This flaw arises when Kubernetes allows a local authenticated attacker to bypass security restrictions, caused by a flaw when using the localhost type for a seccomp profile but specifying an empty profile field. An attacker can bypass the seccomp profile enforcement by sending a specially crafted request. This flaw has been resolved in OADP 1.2.5.

For more details, see (CVE-2023-2431).

CSI restore ended with 'PartiallyFailed' status and PVCs not created

CSI restore ended with PartiallyFailed status. PVCs are not created, pod are in Pending status. This issue has been resolved in OADP 1.2.5.

(OADP-1956)

PodVolumeBackup fails on completed pod volumes

In earlier versions of OADP 1.2, when there is a completed pod that mounted volumes in a namespace used by the Restic podvolumebackup or Velero backup, the backup does not complete successfully. This occurs when defaultVolumesToFsBackup is set to true. This issue has been resolved in OADP 1.2.5.

(OADP-1870)

Known issues

Data Protection Application (DPA) does not reconcile when the credentials secret is updated

Currently, the OADP Operator does not reconcile when you update the cloud-credentials secret. This occurs because there are no OADP specific labels or owner references on the cloud-credentials secret. If you create a cloud-credentials secret with incorrect credentials, such as empty data, the Operator reconciles and creates a backup storage location (BSL) and registry deployment with the empty data. As a result, when you update the cloud-credentials secret with the correct credentials, the OADP Operator does not immediately reconcile to catch the new credentials.

Workaround: Update to OADP 1.3.

(OADP-3327)

OADP 1.2.4 release notes

OpenShift API for Data Protection (OADP) 1.2.4 is a Container Grade Only (CGO) release, released to refresh the health grades of the containers, with no changes to any code in the product itself compared to that of OADP 1.2.3.

Resolved issues

There are no resolved issues in OADP 1.2.4.

Known issues

The OADP 1.2.4 has the following known issue:

Data Protection Application (DPA) does not reconcile when the credentials secret is updated

Currently, the OADP Operator does not reconcile when you update the cloud-credentials secret. This occurs because there are no OADP specific labels or owner references on the cloud-credentials secret. If you create a cloud-credentials secret with incorrect credentials, such as empty data, the Operator reconciles and creates a Backup Storage Location (BSL) and registry deployment with the empty data. As a result, when you update the cloud-credentials secret with the correct credentials, the Operator does not immediately reconcile to catch the new credentials.

Workaround: Update to OADP 1.3.

(OADP-3327)

OADP 1.2.3 release notes

New features

There are no new features in the release of OpenShift API for Data Protection (OADP) 1.2.3.

Resolved issues

The following highlighted issues are resolved in OADP 1.2.3:

Multiple HTTP/2 enabled web servers are vulnerable to a DDoS attack (Rapid Reset Attack)

In previous releases of OADP 1.2, the HTTP/2 protocol was susceptible to a denial of service attack because request cancellation could reset multiple streams quickly. The server had to set up and tear down the streams while not hitting any server-side limit for the maximum number of active streams per connection. This resulted in a denial of service due to server resource consumption. For a list of all OADP issues associated with this CVE, see the following Jira list.

For more information, see CVE-2023-39325 (Rapid Reset Attack).

For a complete list of all issues resolved in the release of OADP 1.2.3, see the list of OADP 1.2.3 resolved issues in Jira.

Known issues

The OADP 1.2.3 has the following known issue:

Data Protection Application (DPA) does not reconcile when the credentials secret is updated

Currently, the OADP Operator does not reconcile when you update the cloud-credentials secret. This occurs because there are no OADP specific labels or owner references on the cloud-credentials secret. If you create a cloud-credentials secret with incorrect credentials, such as empty data, the Operator reconciles and creates a Backup Storage Location (BSL) and registry deployment with the empty data. As a result, when you update the cloud-credentials secret with the correct credentials, the Operator does not immediately reconcile to catch the new credentials.

Workaround: Update to OADP 1.3.

(OADP-3327)

OADP 1.2.2 release notes

New features

There are no new features in the release of OpenShift API for Data Protection (OADP) 1.2.2.

Resolved issues

The following highlighted issues are resolved in OADP 1.2.2:

Restic restore partially failed due to a Pod Security standard

In previous releases of OADP 1.2, OpenShift Container Platform 4.14 enforced a pod security admission (PSA) policy that hindered the readiness of pods during a Restic restore process.

This issue has been resolved in the release of OADP 1.2.2, and also OADP 1.1.6. Therefore, it is recommended that users upgrade to these releases.

For more information, see Restic restore partially failing on OCP 4.14 due to changed PSA policy. (OADP-2094)

Backup of an app with internal images partially failed with plugin panicked error

In previous releases of OADP 1.2, the backup of an application with internal images partially failed with plugin panicked error returned. The backup partially fails with this error in the Velero logs:

time="2022-11-23T15:40:46Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/django-persistent-67a5b83d-6b44-11ed-9cba-902e163f806c logSource="/remote-source/velero/app/pkg/backup/backup.go:413" name=django-psql-persistent
time="2022-11-23T15:40:46Z" level=error msg="Error backing up item" backup=openshift-adp/django-persistent-67a5b83d-6b44-11ed-9cba-902e163f8

This issue has been resolved in OADP 1.2.2. (OADP-1057).

ACM cluster restore was not functioning as expected due to restore order

In previous releases of OADP 1.2, ACM cluster restore was not functioning as expected due to restore order. ACM applications were removed and re-created on managed clusters after restore activation. (OADP-2505)

VM’s using filesystemOverhead failed when backing up and restoring due to volume size mismatch

In previous releases of OADP 1.2, due to storage provider implementation choices, whenever there was a difference between the application persistent volume claims (PVCs) storage request and the snapshot size of the same PVC, VM’s using filesystemOverhead failed when backing up and restoring. This issue has been resolved in the Data Mover of OADP 1.2.2. (OADP-2144)

OADP did not contain an option to set VolSync replication source prune interval

In previous releases of OADP 1.2, there was no option to set the VolSync replication source pruneInterval. (OADP-2052)

Possible pod volume backup failure if Velero was installed in multiple namespaces

In previous releases of OADP 1.2, there was a possibility of pod volume backup failure if Velero was installed in multiple namespaces. (OADP-2409)

Backup Storage Locations moved to unavailable phase when VSL uses custom secret

In previous releases of OADP 1.2, Backup Storage Locations moved to unavailable phase when Volume Snapshot Location used custom secret. (OADP-1737)

For a complete list of all issues resolved in the release of OADP 1.2.2, see the list of OADP 1.2.2 resolved issues in Jira.

Known issues

The following issues have been highlighted as known issues in the release of OADP 1.2.2:

Must-gather command fails to remove ClusterRoleBinding resources

The oc adm must-gather command fails to remove ClusterRoleBinding resources, which are left on cluster due to admission webhook. Therefore, requests for the removal of the ClusterRoleBinding resources are denied. (OADP-27730)

admission webhook "clusterrolebindings-validation.managed.openshift.io" denied the request: Deleting ClusterRoleBinding must-gather-p7vwj is not allowed

For a complete list of all known issues in this release, see the list of OADP 1.2.2 known issues in Jira.

OADP 1.2.1 release notes

New features

There are no new features in the release of OpenShift API for Data Protection (OADP) 1.2.1.

Resolved issues

For a complete list of all issues resolved in the release of OADP 1.2.1, see the list of OADP 1.2.1 resolved issues in Jira.

Known issues

The following issues have been highlighted as known issues in the release of OADP 1.2.1:

DataMover Restic retain and prune policies do not work as expected

The retention and prune features provided by VolSync and Restic are not working as expected. Because there is no working option to set the prune interval on VolSync replication, you have to manage and prune remotely stored backups on S3 storage outside of OADP. For more details, see:

OADP Data Mover is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

For a complete list of all known issues in this release, see the list of OADP 1.2.1 known issues in Jira.

OADP 1.2.0 release notes

The OADP 1.2.0 release notes include information about new features, bug fixes, and known issues.

New features

Resource timeouts

The new resourceTimeout option specifies the timeout duration in minutes for waiting on various Velero resources. This option applies to resources such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default duration is 10 minutes.

AWS S3 compatible backup storage providers

You can back up objects and snapshots on AWS S3 compatible providers.

Technical preview features

Data Mover

The OADP Data Mover enables you to back up Container Storage Interface (CSI) volume snapshots to a remote object store. When you enable Data Mover, you can restore stateful applications using CSI volume snapshots pulled from the object store in case of accidental cluster deletion, cluster failure, or data corruption.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Resolved issues

For a complete list of all issues resolved in this release, see the list of OADP 1.2.0 resolved issues in Jira.

Known issues

The following issues have been highlighted as known issues in the release of OADP 1.2.0:

Multiple HTTP/2 enabled web servers are vulnerable to a DDoS attack (Rapid Reset Attack)

The HTTP/2 protocol is susceptible to a denial of service attack because request cancellation can reset multiple streams quickly. The server has to set up and tear down the streams while not hitting any server-side limit for the maximum number of active streams per connection. This results in a denial of service due to server resource consumption.

It is advised to upgrade to OADP 1.2.3, which resolves this issue.

For more information, see CVE-2023-39325 (Rapid Reset Attack).

An incorrect hostname can be created when changing a hostname in a generated route.

By default, the OpenShift Container Platform cluster makes sure that the openshift.io/host.generated: true annotation is turned on and fills in the field for both the routes that are generated and those that are not generated.

You cannot modify the value for the .spec.host field based on the base domain name of your cluster in the generated and non-generated routes.

If you modify the value for the .spec.host field, it is not possible to restore the default value that was generated by the OpenShift Container Platform cluster. After you restore your OpenShift Container Platform cluster, the Operator resets the value for the field.

Upgrade notes

Always upgrade to the next minor version. Do not skip versions. To update to a later version, upgrade only one channel at a time. For example, to upgrade from OpenShift API for Data Protection (OADP) 1.1 to 1.3, upgrade first to 1.2, then to 1.3.

Changes from OADP 1.1 to 1.2

The Velero server was updated from version 1.9 to 1.11.

In OADP 1.2, the DataProtectionApplication (DPA) configuration dpa.spec.configuration.velero.args has the following changes:

The default-volumes-to-restic field was renamed to default-volumes-to-fs-backup. If you use dpa.spec.configuration.velero.args, you must add it again with the new name to your DPA after upgrading OADP.
The restic-timeout field was renamed to fs-backup-timeout. If you use dpa.spec.configuration.velero.args, you must add it again with the new name to your DPA after upgrading OADP.
The restic daemon set was renamed to node-agent. OADP automatically updates the name of the daemon set.
The custom resource definition resticrepositories.velero.io was renamed to backuprepositories.velero.io.
The custom resource definition resticrepositories.velero.io can be removed from the cluster.

Upgrading steps

Backing up the DPA configuration

You must back up your current DataProtectionApplication (DPA) configuration.

Procedure

Save your current DPA configuration by running the following command:
Example
```
$ oc get dpa -n openshift-adp -o yaml > dpa.orig.backup
```

Upgrading the OADP Operator

Use the following sequence when upgrading the OpenShift API for Data Protection (OADP) Operator.

Procedure

Change your subscription channel for the OADP Operator from stable-1.1 to stable-1.2.
Allow time for the Operator and containers to update and restart.

Additional resources

Converting DPA to the new version

If you use the fields that were updated in the spec.configuration.velero.args stanza, you must configure your DataProtectionApplication (DPA) manifest to use the new parameter names.

Procedure

Click Operators → Installed Operators and select the OADP Operator.
Select Provided APIs, click Create instance in the DataProtectionApplication box.

Click YAML View to display the current DPA parameters.

Example current DPA

spec:
  configuration:
    velero:
      args:
        default-volumes-to-fs-backup: true
        default-restic-prune-frequency: 6000
        fs-backup-timeout: 600
# ...

Update the DPA parameters:
Update the DPA parameter names without changing their values:
1. Change the default-volumes-to-restic key to default-volumes-to-fs-backup.
2. Change the default-restic-prune-frequency key to default-repo-maintain-frequency.
3. Change the restic-timeout key to fs-backup-timeout.
Example updated DPA
```
spec:
  configuration:
    velero:
      args:
        default-volumes-to-fs-backup: true
        default-repo-maintain-frequency: 6000
        fs-backup-timeout: 600
# ...
```
Wait for the DPA to reconcile successfully.

Verifying the upgrade

Use the following procedure to verify the upgrade.

Procedure

Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

$ oc get all -n openshift-adp

Example output

NAME                                                     READY   STATUS    RESTARTS   AGE
pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
pod/restic-9cq4q                                         1/1     Running   0          94s
pod/restic-m4lts                                         1/1     Running   0          94s
pod/restic-pv4kr                                         1/1     Running   0          95s
pod/velero-588db7f655-n842v                              1/1     Running   0          95s

NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s

NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/restic   3         3         3       3            3           <none>          96s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
deployment.apps/velero                              1/1     1            1           96s

NAME                                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
replicaset.apps/velero-588db7f655                              1         1         1       96s

Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'

Example output

{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}

Verify the type is set to Reconciled.

Verify the backup storage location and confirm that the PHASE is Available by running the following command:

$ oc get backupStorageLocation -n openshift-adp

Example output

NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
dpa-sample-1   Available   1s               3d16h   true