$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp
To restore the cluster to a previous state, you must have previously backed up etcd data by creating a snapshot. You will use this snapshot to restore the cluster state.
You can use an etcd backup to restore your cluster to a previous state. This can be used to recover from the following situations:
The cluster has lost the majority of control plane hosts (quorum loss).
An administrator has deleted something critical and must restore to recover the cluster.
Restoring to a previous cluster state is a destructive and destablizing action to take on a running cluster. This should only be used as a last resort. If you are able to retrieve data using the Kubernetes API server, then etcd is available and you should not restore using an etcd backup. |
Restoring etcd effectively takes a cluster back in time and all clients will experience a conflicting, parallel history. This can impact the behavior of watching components like kubelets, Kubernetes controller managers, SDN controllers, and persistent volume controllers.
It can cause Operator churn when the content in etcd does not match the actual content on disk, causing Operators for the Kubernetes API server, Kubernetes controller manager, Kubernetes scheduler, and etcd to get stuck when files on disk conflict with content in etcd. This can require manual actions to resolve the issues.
In extreme cases, the cluster can lose track of persistent volumes, delete critical workloads that no longer exist, reimage machines, and rewrite CA bundles with expired certificates.
You can use a saved etcd backup to restore a previous cluster state or restore a cluster that has lost the majority of control plane hosts.
When you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an OpenShift Container Platform 4.7.2 cluster must use an etcd backup that was taken from 4.7.2. |
Access to the cluster as a user with the cluster-admin
role.
A healthy control plane host to use as the recovery host.
SSH access to control plane hosts.
A backup directory containing both the etcd snapshot and the resources for the static pods, which were from the same backup. The file names in the directory must be in the following formats: snapshot_<datetimestamp>.db
and static_kuberesources_<datetimestamp>.tar.gz
.
For non-recovery control plane nodes, it is not required to establish SSH connectivity or to stop the static pods. You can delete and recreate other non-recovery, control plane machines, one by one. |
Select a control plane host to use as the recovery host. This is the host that you will run the restore operation on.
Establish SSH connectivity to each of the control plane nodes, including the recovery host.
The Kubernetes API server becomes inaccessible after the restore process starts, so you cannot access the control plane nodes. For this reason, it is recommended to establish SSH connectivity to each control plane host in a separate terminal.
If you do not complete this step, you will not be able to access the control plane hosts to complete the restore procedure, and you will be unable to recover your cluster from this state. |
Copy the etcd backup directory to the recovery control plane host.
This procedure assumes that you copied the backup
directory containing the etcd snapshot and the resources for the static pods to the /home/core/
directory of your recovery control plane host.
Stop the static pods on any other control plane nodes.
It is not required to manually stop the pods on the recovery host. The recovery script will stop the pods on the recovery host. |
Access a control plane host that is not the recovery host.
Move the existing etcd pod file out of the kubelet manifest directory:
$ sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp
Verify that the etcd pods are stopped.
$ sudo crictl ps | grep etcd | grep -v operator
The output of this command should be empty. If it is not empty, wait a few minutes and check again.
Move the existing Kubernetes API server pod file out of the kubelet manifest directory:
$ sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp
Verify that the Kubernetes API server pods are stopped.
$ sudo crictl ps | grep kube-apiserver | grep -v operator
The output of this command should be empty. If it is not empty, wait a few minutes and check again.
Move the etcd data directory to a different location:
$ sudo mv /var/lib/etcd/ /tmp
Repeat this step on each of the other control plane hosts that is not the recovery host.
Access the recovery control plane host.
If the cluster-wide proxy is enabled, be sure that you have exported the NO_PROXY
, HTTP_PROXY
, and HTTPS_PROXY
environment variables.
You can check whether the proxy is enabled by reviewing the output of |
Run the restore script on the recovery control plane host and pass in the path to the etcd backup directory:
$ sudo -E /usr/local/bin/cluster-restore.sh /home/core/backup
...stopping kube-scheduler-pod.yaml
...stopping kube-controller-manager-pod.yaml
...stopping etcd-pod.yaml
...stopping kube-apiserver-pod.yaml
Waiting for container etcd to stop
.complete
Waiting for container etcdctl to stop
.............................complete
Waiting for container etcd-metrics to stop
complete
Waiting for container kube-controller-manager to stop
complete
Waiting for container kube-apiserver to stop
..........................................................................................complete
Waiting for container kube-scheduler to stop
complete
Moving etcd data-dir /var/lib/etcd/member to /var/lib/etcd-backup
starting restore-etcd static pod
starting kube-apiserver-pod.yaml
static-pod-resources/kube-apiserver-pod-7/kube-apiserver-pod.yaml
starting kube-controller-manager-pod.yaml
static-pod-resources/kube-controller-manager-pod-7/kube-controller-manager-pod.yaml
starting kube-scheduler-pod.yaml
static-pod-resources/kube-scheduler-pod-8/kube-scheduler-pod.yaml
The restore process can cause nodes to enter the |
Check the nodes to ensure they are in the Ready
state.
Run the following command:
$ oc get nodes -w
NAME STATUS ROLES AGE VERSION
host-172-25-75-28 Ready master 3d20h v1.24.0
host-172-25-75-38 Ready infra,worker 3d20h v1.24.0
host-172-25-75-40 Ready master 3d20h v1.24.0
host-172-25-75-65 Ready master 3d20h v1.24.0
host-172-25-75-74 Ready infra,worker 3d20h v1.24.0
host-172-25-75-79 Ready worker 3d20h v1.24.0
host-172-25-75-86 Ready worker 3d20h v1.24.0
host-172-25-75-98 Ready infra,worker 3d20h v1.24.0
It can take several minutes for all nodes to report their state.
If any nodes are in the NotReady
state, log in to the nodes and remove all of the PEM files from the /var/lib/kubelet/pki
directory on each node. You can SSH into the nodes or use the terminal window in the web console.
$ ssh -i <ssh-key-path> core@<master-hostname>
pki
directorysh-4.4# pwd
/var/lib/kubelet/pki
sh-4.4# ls
kubelet-client-2022-04-28-11-24-09.pem kubelet-server-2022-04-28-11-24-15.pem
kubelet-client-current.pem kubelet-server-current.pem
Restart the kubelet service on all control plane hosts.
From the recovery host, run the following command:
$ sudo systemctl restart kubelet.service
Repeat this step on all other control plane hosts.
Approve the pending CSRs:
Get the list of current CSRs:
$ oc get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION csr-2s94x 8m3s kubernetes.io/kubelet-serving system:node:<node_name> Pending (1) csr-4bd6t 8m3s kubernetes.io/kubelet-serving system:node:<node_name> Pending (1) csr-4hl85 13m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending (2) csr-zhhhp 3m8s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending (2) ...
1 | A pending kubelet service CSR (for user-provisioned installations). |
2 | A pending node-bootstrapper CSR. |
Review the details of a CSR to verify that it is valid:
$ oc describe csr <csr_name> (1)
1 | <csr_name> is the name of a CSR from the list of current CSRs. |
Approve each valid node-bootstrapper
CSR:
$ oc adm certificate approve <csr_name>
For user-provisioned installations, approve each valid kubelet service CSR:
$ oc adm certificate approve <csr_name>
Verify that the single member control plane has started successfully.
From the recovery host, verify that the etcd container is running.
$ sudo crictl ps | grep etcd | grep -v operator
3ad41b7908e32 36f86e2eeaaffe662df0d21041eb22b8198e0e58abeeae8c743c3e6e977e8009 About a minute ago Running etcd 0 7c05f8af362f0
From the recovery host, verify that the etcd pod is running.
$ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd
If you attempt to run
|
NAME READY STATUS RESTARTS AGE
etcd-ip-10-0-143-125.ec2.internal 1/1 Running 1 2m47s
If the status is Pending
, or the output lists more than one running etcd pod, wait a few minutes and check again.
Repeat this step for each lost control plane host that is not the recovery host.
Perform the following step only if you are using |
Restart the Open Virtual Network (OVN) Kubernetes pods on all the hosts.
Remove the northbound database (nbdb) and southbound database (sbdb). Access the recovery host and the remaining control plane nodes by using Secure Shell (SSH) and run the following command:
$ sudo rm -f /var/lib/ovn/etc/*.db
Delete all OVN-Kubernetes control plane pods by running the following command: