Troubleshoot Ceph RBD integration

If Compute (nova), Block Storage (cinder), or Image (glance) service integration with Ceph RBD fails, use incremental troubleshooting to isolate the issue. The following steps focus on Image service integration, but you can adapt them for other services.

About this task
Copy link

If you discover the cause of your issue before completing this procedure, it is not necessary to do any subsequent steps. You can exit this procedure and resolve the issue.

Procedure
Copy link

  1. Determine if any parts of the control plane are not properly deployed by assessing whether the Ready condition is not True: 
    $ oc get -n openstack OpenStackControlPlane \
  -o jsonpath="{range .items[0].status.conditions[?(@.status!='True')]}{.type} is {.status} due to {.message}{'\n'}{end}"
    1. If you identify a service that is not properly deployed, check the status of the service.

      The following example checks the status of the Compute service:

      $ oc get -n openstack Nova/nova \
  -o jsonpath="{range .status.conditions[?(@.status!='True')]}{.type} is {.status} due to {.message}{'\n'}{end}"
      • You can check the status of all deployed services: 
        $ oc get pods -n openstack
      • You can check the logs of a specific service: 
        $ oc logs -n openstack <service_pod_name>
        • Replace <service_pod_name> with the name of the service pod you want to check.
    2. If you identify an operator that is not properly deployed, check the status of the operator: 
      $ oc get pods -n openstack-operators -lopenstack.org/operator-name
      • You can check the operator logs: 
        $ oc logs -n openstack-operators -lopenstack.org/operator-name=<operator_name>
  2. Check the Status of the data plane deployment: 
    $ oc get -n openstack OpenStackDataPlaneDeployment
    • If the Status of the data plane deployment is False, check the logs of the associated Ansible job: 
      $ oc logs -n openstack job/<ansible_job_name>
      • Replace <ansible_job_name> with the name of the associated job. The job name is listed in the Message field of the oc get -n openstack OpenStackDataPlaneDeployment command output.
  3. Check the Status of the data plane node set deployment: 
    $ oc get -n openstack OpenStackDataPlaneNodeSet
    • If the Status of the data plane node set deployment is False, check the logs of the associated Ansible job: 
      $ oc logs -n openstack job/<ansible_job_name>
      • Replace <ansible_job_name> with the name of the associated job. It is listed in the Message field of the oc get -n openstack OpenStackDataPlaneNodeSet command output.
  4. If any pods are in the CrashLookBackOff state, you can duplicate them for troubleshooting purposes: 
    $ oc debug <pod_name>
    • Replace <pod_name> with the name of the pod to duplicate.
  5. Optional: You can route traffic to the duplicate pod during the debug process: 
    $ oc debug <pod_name> --keep-labels=true
  6. Optional: You can use the oc debug command in the following object debugging activities:
    • To run /bin/sh on a container other than the first one, the command’s default behavior, using the command form oc debug -container <pod_name> <container_name>. This is useful for pods such as the API where the first container is tailing a file and the second container is the one you want to debug. If you use this command form, you must first use the command oc get pods | grep <search_string> to find the container name.
    • To create any resource that creates pods such as Deployments, StatefulSets, and Nodes, use the command form oc debug <resource_type>/<resource_name>. An example of creating a StatefulSet would be oc debug StatefulSet/cinder-scheduler.
  7. Connect to the pod and confirm that the ceph.client.openstack.keyring and ceph.conf files are present in the /etc/ceph directory. 
    $ oc rsh <pod_name>
    • Replace <pod_name> with the name of the applicable pod.
    • If the Ceph configuration files are missing, check the extraMounts parameter in your OpenStackControlPlane CR.
  8. Confirm the pod has a network connection to the Red Hat Ceph Storage cluster by connecting to the IP and port of a Ceph Monitor from the pod. The IP and port information is located in /etc/ceph.conf.

    The following is an example of this process:

    $ oc get pods | grep glance | grep external-api-0
glance-06f7a-default-external-api-0                               3/3     Running     0              2d3h
$ oc debug --container glance-api glance-06f7a-default-external-api-0
Starting pod/glance-06f7a-default-external-api-0-debug-p24v9, command was: /usr/bin/dumb-init --single-child -- /bin/bash -c /usr/local/bin/kolla_set_configs && /usr/local/bin/kolla_start
Pod IP: 192.168.25.50
If you don't see a command prompt, try pressing enter.
sh-5.1# cat /etc/ceph/ceph.conf
# Ansible managed

[global]

fsid = 63bdd226-fbe6-5f31-956e-7028e99f1ee1
mon host = [v2:192.168.122.100:3300/0,v1:192.168.122.100:6789/0],[v2:192.168.122.102:3300/0,v1:192.168.122.102:6789/0],[v2:192.168.122.101:3300/0,v1:192.168.122.101:6789/0]


[client.libvirt]
admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/ceph/qemu-guest-$pid.log

sh-5.1# python3
Python 3.9.19 (main, Jul 18 2024, 00:00:00)
[GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket()
>>> ip="192.168.122.100"
>>> port=3300
>>> s.connect((ip,port))
>>>
  9. Optional: If you cannot connect to a Ceph Monitor, troubleshoot the network connection between the cluster and pod. The previous example uses a Python socket to connect to the IP and port of the Red Hat Ceph Storage cluster from the ceph.conf file.
    • There are two potential outcomes from the execution of the s.connect((ip,port)) function:
      • If the command executes successfully and there is no error similar to the following example, the network connection between the pod and cluster is functioning correctly. If the connection is functioning correctly, the command execution will provide no return value at all.
      • If the command takes a long time to run and returns an error similar to the following example, the network connection between the pod and cluster is not functioning correctly. It should be investigated further to troubleshoot the connection. 
        Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TimeoutError: [Errno 110] Connection timed out
  10. Examine the cephx key as shown in the following example: 
    bash-5.1$ cat /etc/ceph/ceph.client.openstack.keyring
[client.openstack]
   key = "<redacted>"
   caps mgr = allow *
   caps mon = profile rbd
   caps osd = profile rbd pool=vms, profile rbd pool=volumes, profile rbd pool=backups, profile rbd pool=images
bash-5.1$
  11. List the contents of a pool from the caps osd parameter as shown in the following example: 
    $ /usr/bin/rbd --conf /etc/ceph/ceph.conf \
--keyring /etc/ceph/ceph.client.openstack.keyring \
--cluster ceph --id openstack \
ls -l -p <pool_name> | wc -l
    • Replace <pool_name> with the name of the required Red Hat Ceph Storage pool.
    • If this command returns the number 0 or greater, the cephx key provides adequate permissions to connect to, and read information from, the Red Hat Ceph Storage cluster.
    • If this command does not complete but network connectivity to the cluster was confirmed, work with the Ceph administrator to obtain the correct cephx keyring.
    • Check if there is an MTU mismatch on the Storage network. If the network is using jumbo frames (an MTU value of 9000), all switch ports between servers using the interface must be updated to support jumbo frames. If this change is not made on the switch, problems can occur at the Ceph application layer. Verify all hosts using the network can communicate at the required MTU with a command such as ping -M do -s 8972 <ip_address>.
  12. Send test data to the images pool on the Ceph cluster.

    The following is an example of performing this task:

    # DATA=$(date | md5sum | cut -c-12)
# POOL=images
# RBD="/usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack"
# $RBD create --size 1024 $POOL/$DATA
    Tip

    It is possible to be able to read data from the cluster but not have permissions to write data to it, even if write permission was granted in the cephx keyring. If you have write permissions, but you cannot write data to the cluster, the cluster might be overloaded and not able to write new data.

    In the example, the rbd command did not complete successfully and was canceled. It was subsequently confirmed the cluster itself did not have the resources to write new data. The issue was resolved on the cluster itself. There was nothing incorrect with the client configuration.

Troubleshoot Ceph clients
Copy link

Put Red Hat OpenStack Services on OpenShift (RHOSO) Red Hat Ceph Storage clients in debug mode to troubleshoot their operation.

Procedure
Copy link

  1. Locate the Red Hat Ceph Storage configuration file mapped in the Red Hat OpenShift secret created in Create a Ceph storage secret.
  2. Modify the contents of the configuration file to include troubleshooting-related configuration.

    The following is an example of troubleshooting-related configuration:

    [client.openstack]
admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/guest-$pid.log
debug ms = 1
debug rbd = 20
log to file = true
    Note

    For more information about troubleshooting, see the Red Hat Ceph Storage Troubleshooting Guide:
  3. Update the secret with the new content.