Reboot data plane nodes

Reboot data plane nodes after updates or for troubleshooting. The reboot method and configuration options differ by node type, such as Compute or Object Storage.

Configure the data plane Compute node reboot behavior
Copy link

Configure how the data plane shuts down and restarts instances on Compute nodes during a reboot. This applies to instances that are not migrated off the host before the reboot.

Before you begin
Copy link

  • You are logged on to a workstation that has access to the Red Hat OpenShift Container Platform (RHOCP) cluster as a user with cluster-admin privileges.

Procedure
Copy link

  1. Open the nova-extra-config.yaml definition file for the default Compute service (nova) ConfigMap custom resource (CR) named nova-extra-config and add the following configuration: 
    apiVersion: v1
kind: ConfigMap
metadata:
  name: nova-extra-config
  namespace: openstack
data:
   30-nova-reboot.conf: |
   [DEFAULT]
   resume_guests_state_on_host_boot=true
   shutdown_timeout=300
    • resume_guests_state_on_host_boot: Set to true to return instances to the same state on the Compute node after reboot. When set to False, the instances remain down and you must start them manually.
    • shutdown_timeout: Specify the number of seconds to wait for an instance to perform a controlled, clean shut down before being powered off and rebooted. Do not set this value to 0. A value of 0 (zero) means that the instance is powered off immediately with no opportunity for instance operating system clean-up. The default value is 60.
  2. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR. The OpenStackDataPlaneDeployment CR configures the Compute nodes on the data plane that run the default nova service: 
    apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: <node_set_deployment_name>
spec:
  servicesOverride:
    - nova
  nodeSets:
    - <nodeSet_name>

    where:

    <node_set_deployment_name>
    Specifies the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
    <nodeSet_name>
    Specifies the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
    Tip
    Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that reflect the purpose of the node sets in the deployment.
  3. Save the OpenStackDataPlaneDeployment CR deployment file.
  4. Deploy the modified OpenStackDataPlaneNodeSet CRs: 
    $ oc create -f openstack_data_plane_deploy.yaml -n openstack

    You can view the Ansible logs while the deployment executes:

    $ oc get pod -l app=openstackansibleee -w
$ oc logs -l app=openstackansibleee -f --max-log-requests 10
  5. Verify that the modified OpenStackDataPlaneNodeSet CRs are deployed: 
    $ oc get openstackdataplanedeployment -n openstack
NAME             	STATUS   MESSAGE
openstack-data-plane   True     Setup Complete


$ oc get openstackdataplanenodeset -n openstack
NAME             	STATUS   MESSAGE
openstack-data-plane   True     NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshoot data plane creation and deployment.

Reboot Object Storage service (swift) nodes
Copy link

Reboot Object Storage service (swift) nodes one at a time to ensure cluster availability. Repeat this procedure for every Object Storage node in your cluster.

Procedure
Copy link

  1. Log in to an Object Storage node.
  2. Reboot the node: 
    $ sudo reboot
  3. Wait until the node boots.
  4. Repeat the reboot for each Object Storage node in the cluster.

Reboot updated Compute nodes
Copy link

Check which updated Compute nodes require a reboot. You can reboot the nodes any time after completing the minor update. For data plane development kit (DPDK) implementations, the old kernel and Open vSwitch (OVS) remain active until reboot. Migrate instances to ensure minimal downtime.

Before you begin
Copy link

  • You have decided whether to migrate instances to another Compute node before you start the reboot. Note
    • If you have a Multi-RHEL environment, and you want to migrate virtual machines from a Compute node that is running RHEL 9.4 or 9.6 to a Compute node that is running RHEL 9.2, only cold migration is supported.
    • If you cannot migrate the instances, you can set the shutdown_timeout configuration option to control the state of the instances after the Compute node reboots. This option determines the number of seconds to wait for an instance to perform a clean shutdown. The default value is 60.

About this task
Copy link

Note
If you enabled kernel live patching (kpatch) on a set of Compute nodes, you are not required to reboot those nodes. For more information about enabling kpatch, see Enabling kernel live patching on Compute nodes.

Procedure
Copy link

  1. Confirm which updated nodes need a reboot: 
    $ oc logs jobs/reboot-os-<deployment_name>-<nodeSet_name>
    • Replace <deployment_name> with the name of the deployment that includes your Compute nodes.
    • Replace <nodeSet_name> with the names of the node sets that you need to check.

      The command shows the following output if a reboot is required: Reboot is required but was not started. Edpm_reboot_strategy is set to never or this is already deployed machine. Reboot has to be planned. To start reboot set edpm_reboot_strategy to force.

  2. Open a remote shell connection to the OpenStackClient pod: 
    $ oc rsh -n openstack openstackclient
  3. Retrieve a list of your Compute nodes to identify the hostname of the nodes that require a reboot: 
    $ openstack compute service list
  4. Disable the Compute service on the Compute node that you need to reboot: 
    $ openstack compute service set <hostname> nova-compute --disable
    • Replace <hostname> with the hostname of the Compute node on which you are disabling the service.
  5. If Red Hat Ceph Storage is also hosted on the Compute node, stop Ceph services but do not shut down hosts.
    • In Red Hat Ceph Storage 7 and 8, perform steps 1 to 14 in the procedure "Powering down and rebooting the cluster using Ceph Orchestrator" in the Red Hat Ceph Storage Administration Guide. Do not reboot the cluster as instructed in the Administration Guide. The cluster is rebooted later in this procedure.
    • In Red Hat Ceph Storage 9, perform steps 1 to 12 in the procedure "Powering down the Red Hat Ceph Storage cluster" in the Red Hat Ceph Storage Administration Guide. Do not perform steps 13-15, which shut down the OSD hosts, monitor hosts, and remaining standalone hosts. The Compute node is rebooted later in this procedure.
  6. List all instances on the Compute node: 
    $ openstack server list --host <hostname> --all-projects
  7. Optional: If you decide to migrate the instances to another Compute node, for example, if you plan to reboot nodes that include running workloads, run the following command: 
    $ openstack server migrate --live-migration --host <target_host>  --wait <instance_id>
    • Replace <instance_id> with your instance ID.
    • Replace <target_host> with the host that you are migrating the instance to.
      1. Wait until migration completes.
      2. Confirm that the migration was successful: 
        $ openstack server list --host <hostname> --all-projects
      3. Continue to migrate instances until none remain on the Compute node.
  8. Exit the OpenStackClient pod: 
    $ exit
  9. Create an OpenStackDataPlaneDeployment CR file to reboot the nodes: 
    apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: openstack-edpm-ipam-reboot
  namespace: openstack
spec:
  nodeSets:
  - <nodeSet_name>
  servicesOverride:
  - reboot-os
  ansibleExtraVars:
    edpm_reboot_strategy: force
  ansibleLimit: <node_hostname>,...,<node_hostname>
    • spec.nodeSets lists the OpenStackDataPlaneNodeSet CRs that contain the nodes that you are rebooting.
    • spec.servicesOverride specifies the reboot-os as the only service to run.
    • spec.ansibleExtraVars reboots all the nodes in the node set at the same time.
    • Optional: spec.ansibleLimit lists the individual nodes in the node set to reboot. If not set, all the nodes in the node set are rebooted at the same time.
    • <node_hostname> specifies the hostname of the node to reboot.
  10. Save the OpenStackDataPlaneDeployment CR file.
  11. Apply the OpenStackDataPlaneDeployment CR to your environment: 
    $ oc apply -f <openstack_dataplane_deployment_CR_file> -n openstack

    where:

    <openstack_dataplane_deployment_CR_file>
    Specifies the name of the OpenStackDataPlaneDeployment CR file.
  12. Verify that the openstack-edpm-ipam-reboot deployment completed: 
    $ oc get openstackdataplanedeployment
NAME                                    STATUS   MESSAGE
openstack-edpm-deployment-ipam-reboot   True     Setup complete

    If the deployment fails, see Troubleshoot data plane creation and deployment.

  13. Re-enable the Compute node: 
    $ oc rsh openstackclient -n openstack
$ openstack compute service set <hostname>  nova-compute --enable
  14. If Red Hat Ceph Storage is also hosted on the Compute node, perform steps 1 to 7 in the procedure "Powering down and rebooting the cluster using Ceph Orchestrator" in the Red Hat Ceph Storage Administration Guide:
  15. Check that the Compute node is enabled: 
    $ openstack compute service list

Compute hostname change detection troubleshooting
Copy link

If the Compute service (nova) detects a hostname change on startup, it aborts to prevent data inconsistency. Identify and resolve the cause of the hostname change to recover.

You receive the following error message:

Found 7 instances on hypervisor but no ComputeNode record exists with host compute01.example.com node compute01.example.com; potential rename detected - aborting startup

To recover from this error, you must know the reason for the change of the hostname. Here are the three possible reasons for the change of the hostname:

  • You have changed the name of the Compute host.
  • The DNS is not working correctly and has assigned a different name to the Compute host.
  • There is another deployment issue causing a Compute hostname to be incorrect.

When you resolve the issue, you must restart the Compute service.

Note
The nova-libvirt container contains information about the hostname. If you resolve an issue with a Compute hostname, restart both the nova-libvirt and nova-compute containers to ensure that all components contain the correct name.

For further assistance, contact the Red Hat Technical Support Team.