1. Documentation
    2. history bug_report picture_as_pdf
×

Restoring to a previous cluster state

To restore the cluster to a previous state, you must have previously backed up etcd data by creating a snapshot. You will use this snapshot to restore the cluster state.

About restoring cluster state

You can use an etcd backup to restore your cluster to a previous state. This can be used to recover from the following situations:

  • The cluster has lost the majority of control plane hosts (quorum loss).

  • An administrator has deleted something critical and must restore to recover the cluster.

Restoring to a previous cluster state is a destructive and destablizing action to take on a running cluster. This should only be used as a last resort.

If you are able to retrieve data using the Kubernetes API server, then etcd is available and you should not restore using an etcd backup.

Restoring etcd effectively takes a cluster back in time and all clients will experience a conflicting, parallel history. This can impact the behavior of watching components like kubelets, Kubernetes controller managers, SDN controllers, and persistent volume controllers.

It can cause Operator churn when the content in etcd does not match the actual content on disk, causing Operators for the Kubernetes API server, Kubernetes controller manager, Kubernetes scheduler, and etcd to get stuck when files on disk conflict with content in etcd. This can require manual actions to resolve the issues.

In extreme cases, the cluster can lose track of persistent volumes, delete critical workloads that no longer exist, reimage machines, and rewrite CA bundles with expired certificates.

Restoring to a previous cluster state

You can use a saved etcd backup to restore a previous cluster state or restore a cluster that has lost the majority of control plane hosts.

When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an OpenShift Container Platform 4.7.2 cluster must use an etcd backup that was taken from 4.7.2.
Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

  • A healthy control plane host to use as the recovery host.

  • SSH access to control plane hosts.

  • A backup directory containing both the etcd snapshot and the resources for the static pods, which were from the same backup. The file names in the directory must be in the following formats: snapshot_<datetimestamp>.db and static_kuberesources_<datetimestamp>.tar.gz.

For non-recovery control plane nodes, it is not required to establish SSH connectivity or to stop the static pods. You can delete and recreate other non-recovery, control plane machines, one by one.
Procedure

  1. Select a control plane host to use as the recovery host. This is the host that you will run the restore operation on.

  2. Establish SSH connectivity to each of the control plane nodes, including the recovery host.

    The Kubernetes API server becomes inaccessible after the restore process starts, so you cannot access the control plane nodes. For this reason, it is recommended to establish SSH connectivity to each control plane host in a separate terminal.

    If you do not complete this step, you will not be able to access the control plane hosts to complete the restore procedure, and you will be unable to recover your cluster from this state.

  3. Copy the etcd backup directory to the recovery control plane host.

    This procedure assumes that you copied the backup directory containing the etcd snapshot and the resources for the static pods to the /home/core/ directory of your recovery control plane host.

  4. Stop the static pods on any other control plane nodes.

    It is not required to manually stop the pods on the recovery host. The recovery script will stop the pods on the recovery host.

    1. Access a control plane host that is not the recovery host.

    2. Move the existing etcd pod file out of the kubelet manifest directory:

      $ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp

    3. Verify that the etcd pods are stopped.

      $ sudo crictl ps | grep etcd | grep -v operator

      The output of this command should be empty. If it is not empty, wait a few minutes and check again.

    4. Move the existing Kubernetes API server pod file out of the kubelet manifest directory:

      $ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp

    5. Verify that the Kubernetes API server pods are stopped.

      $ sudo crictl ps | grep kube-apiserver | grep -v operator

      The output of this command should be empty. If it is not empty, wait a few minutes and check again.

    6. Move the etcd data directory to a different location:

      $ sudo mv /var/lib/etcd/ /tmp

    7. Repeat this step on each of the other control plane hosts that is not the recovery host.

  5. Access the recovery control plane host.

  6. If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXY, HTTP_PROXY, and HTTPS_PROXY environment variables.

    You can check whether the proxy is enabled by reviewing the output of oc get proxy cluster -o yaml. The proxy is enabled if the httpProxy, httpsProxy, and noProxy fields have values set.

  7. Run the restore script on the recovery control plane host and pass in the path to the etcd backup directory:

    $ sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup
    Example script output
    ...stopping kube-scheduler-pod.yaml
...stopping kube-controller-manager-pod.yaml
...stopping etcd-pod.yaml
...stopping kube-apiserver-pod.yaml
Waiting for container etcd to stop
.complete
Waiting for container etcdctl to stop
.............................complete
Waiting for container etcd-metrics to stop
complete
Waiting for container kube-controller-manager to stop
complete
Waiting for container kube-apiserver to stop
..........................................................................................complete
Waiting for container kube-scheduler to stop
complete
Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup
starting restore-etcd static pod
starting kube-apiserver-pod.yaml
static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
starting kube-controller-manager-pod.yaml
static-pod-resources/kube-controller-manager-pod-7/kube-controller-manager-pod.yaml
starting kube-scheduler-pod.yaml
static-pod-resources/kube-scheduler-pod-8/kube-scheduler-pod.yaml

    The restore process can cause nodes to enter the NotReady state if the node certificates were updated after the last etcd backup.

  8. Check the nodes to ensure they are in the Ready state.

    1. Run the following command:

      $ oc get nodes -w
      Sample output
      NAME                STATUS  ROLES          AGE     VERSION
host-172-25-75-28   Ready   master         3d20h   v1.24.0
host-172-25-75-38   Ready   infra,worker   3d20h   v1.24.0
host-172-25-75-40   Ready   master         3d20h   v1.24.0
host-172-25-75-65   Ready   master         3d20h   v1.24.0
host-172-25-75-74   Ready   infra,worker   3d20h   v1.24.0
host-172-25-75-79   Ready   worker         3d20h   v1.24.0
host-172-25-75-86   Ready   worker         3d20h   v1.24.0
host-172-25-75-98   Ready   infra,worker   3d20h   v1.24.0

      It can take several minutes for all nodes to report their state.

    2. If any nodes are in the NotReady state, log in to the nodes and remove all of the PEM files from the /var/lib/kubelet/pki directory on each node. You can SSH into the nodes or use the terminal window in the web console.

      $  ssh -i <ssh-key-path> core@<master-hostname>
      Sample pki directory
      sh-4.4# pwd
/var/lib/kubelet/pki
sh-4.4# ls
kubelet-client-2022-04-28-11-24-09.pem  kubelet-server-2022-04-28-11-24-15.pem
kubelet-client-current.pem              kubelet-server-current.pem

  9. Restart the kubelet service on all control plane hosts.

    1. From the recovery host, run the following command:

      $ sudo systemctl restart kubelet.service

    2. Repeat this step on all other control plane hosts.

  10. Approve the pending CSRs:

    1. Get the list of current CSRs:

      $ oc get csr
      Example output
      NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
csr-2s94x   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending (1)
csr-4bd6t   8m3s   kubernetes.io/kubelet-serving                 system:node:<node_name>                                                     Pending (1)
csr-4hl85   13m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending (2)
csr-zhhhp   3m8s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending (2)
...
      1 A pending kubelet service CSR (for user-provisioned installations).
      2 A pending node-bootstrapper CSR.

    2. Review the details of a CSR to verify that it is valid:

      $ oc describe csr <csr_name> (1)
      1 <csr_name> is the name of a CSR from the list of current CSRs.

    3. Approve each valid node-bootstrapper CSR:

      $ oc adm certificate approve <csr_name>

    4. For user-provisioned installations, approve each valid kubelet service CSR:

      $ oc adm certificate approve <csr_name>

  11. Verify that the single member control plane has started successfully.

    1. From the recovery host, verify that the etcd container is running.

      $ sudo crictl ps | grep etcd | grep -v operator
      Example output
      3ad41b7908e32       36f86e2eeaaffe662df0d21041eb22b8198e0e58abeeae8c743c3e6e977e8009                                                         About a minute ago   Running             etcd                                          0                   7c05f8af362f0

    2. From the recovery host, verify that the etcd pod is running.

      $ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd

      If you attempt to run oc login prior to running this command and receive the following error, wait a few moments for the authentication controllers to start and try again.

      Unable to connect to the server: EOF
      Example output
      NAME                                             READY   STATUS      RESTARTS   AGE
etcd-ip-10-0-143-125.ec2.internal                1/1     Running     1          2m47s

      If the status is Pending, or the output lists more than one running etcd pod, wait a few minutes and check again.

    3. Repeat this step for each lost control plane host that is not the recovery host.

      Perform the following step only if you are using OVNKubernetes Container Network Interface (CNI) plugin.

  12. Restart the Open Virtual Network (OVN) Kubernetes pods on all the hosts.

    1. Remove the northbound database (nbdb) and southbound database (sbdb). Access the recovery host and the remaining control plane nodes by using Secure Shell (SSH) and run the following command:

      $ sudo rm -f /var/lib/ovn/etc/*.db

    2. Delete all OVN-Kubernetes control plane pods by running the following command: