Soon, all Red Hat OpenShift documentation will only be available on docs.redhat.com, the official location of all Red Hat product documentation. Get familiar with the updated experience today!
  1. Documentation
    2. history bug_report picture_as_pdf
×

Collecting data for Red Hat Support

When you submit a support case to Red Hat Support, it is helpful to provide debugging information for OpenShift Container Platform and OpenShift Virtualization by using the following tools:

must-gather tool

The must-gather tool collects diagnostic information, including resource definitions and service logs.

Prometheus

Prometheus is a time-series database and a rule evaluation engine for metrics. Prometheus sends alerts to Alertmanager for processing.

Alertmanager

The Alertmanager service handles alerts received from Prometheus. The Alertmanager is also responsible for sending the alerts to external notification systems.

For information about the OpenShift Container Platform monitoring stack, see About OpenShift Container Platform monitoring.

Collecting data about your environment

Collecting data about your environment minimizes the time required to analyze and determine the root cause.

Prerequisites
Procedure

  1. Collect must-gather data for the cluster.

  2. Collect must-gather data for Red Hat OpenShift Data Foundation, if necessary.

  3. Collect must-gather data for OpenShift Virtualization.

  4. Collect Prometheus metrics for the cluster.

Collecting data about virtual machines

Collecting data about malfunctioning virtual machines (VMs) minimizes the time required to analyze and determine the root cause.

Prerequisites
Procedure

  1. Collect must-gather data for the VMs using the /usr/bin/gather script.

  2. Collect screenshots of VMs that have crashed before you restart them.

  3. Collect memory dumps from VMs before remediation attempts.

  4. Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network.

Using the must-gather tool for OpenShift Virtualization

You can collect data about OpenShift Virtualization resources by running the must-gather command with the OpenShift Virtualization image.

The default data collection includes information about the following resources:

  • OpenShift Virtualization Operator namespaces, including child objects

  • OpenShift Virtualization custom resource definitions

  • Namespaces that contain virtual machines

  • Basic virtual machine definitions

Instance types information is not currently collected by default; you can, however, run a command to optionally collect it.

Procedure

  • Run the following command to collect data about OpenShift Virtualization:

    $ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.16.4 \
  -- /usr/bin/gather

must-gather tool options

You can run the oc adm must-gather command to collect must gather images for all the Operators and products deployed on your cluster without the need to explicitly specify the required images. Alternatively, you can specify a combination of scripts and environment variables for the following options:

  • Collecting detailed virtual machine (VM) information from a namespace

  • Collecting detailed information about specified VMs

  • Collecting image, image-stream, and image-stream-tags information

  • Limiting the maximum number of parallel processes used by the must-gather tool

Parameters

Environment variables

You can specify environment variables for a compatible script.

NS=<namespace_name>

Collect virtual machine information, including virt-launcher pod details, from the namespace that you specify. The VirtualMachine and VirtualMachineInstance CR data is collected for all namespaces.

VM=<vm_name>

Collect details about a particular virtual machine. To use this option, you must also specify a namespace by using the NS environment variable.

PROS=<number_of_processes>

Modify the maximum number of parallel processes that the must-gather tool uses. The default value is 5.

Using too many parallel processes can cause performance issues. Increasing the maximum number of parallel processes is not recommended.
Scripts

Each script is compatible only with certain environment variable combinations.

/usr/bin/gather

Use the default must-gather script, which collects cluster data from all namespaces and includes only basic VM information. This script is compatible only with the PROS variable.

/usr/bin/gather --vms_details

Collect VM log files, VM definitions, control-plane logs, and namespaces that belong to OpenShift Virtualization resources. Specifying namespaces includes their child objects. If you use this parameter without specifying a namespace or VM, the must-gather tool collects this data for all VMs in the cluster. This script is compatible with all environment variables, but you must specify a namespace if you use the VM variable.

/usr/bin/gather --images

Collect image, image-stream, and image-stream-tags custom resource information. This script is compatible only with the PROS variable.

/usr/bin/gather --instancetypes

Collect instance types information. This information is not currently collected by default; you can, however, optionally collect it.

Usage and examples

Environment variables are optional. You can run a script by itself or with one or more compatible environment variables.

Table 1. Compatible parameters
Script Compatible environment variable

/usr/bin/gather

* PROS=<number_of_processes>

/usr/bin/gather --vms_details

* For a namespace: NS=<namespace_name>

* For a VM: VM=<vm_name> NS=<namespace_name>

* PROS=<number_of_processes>

/usr/bin/gather --images

* PROS=<number_of_processes>
Syntax

To collect must-gather logs for all Operators and products on your cluster in a single pass, run the following command:

$ oc adm must-gather --all-images

If you need to pass additional parameters to individual must-gather images, use the following command:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.16.4 \
  -- <environment_variable_1> <environment_variable_2> <script_name>
Default data collection parallel processes

By default, five processes run in parallel.

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.16.4 \
  -- PROS=5 /usr/bin/gather (1)
1 You can modify the number of parallel processes by changing the default.
Detailed VM information

The following command collects detailed VM information for the my-vm VM in the mynamespace namespace:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.16.4 \
  -- NS=mynamespace VM=my-vm /usr/bin/gather --vms_details (1)
1 The NS environment variable is mandatory if you use the VM environment variable.
Image, image-stream, and image-stream-tags information

The following command collects image, image-stream, and image-stream-tags information from the cluster:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.16.4 \
  /usr/bin/gather --images
Instance types information

The following command collects instance types information from the cluster:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.16.4 \
  /usr/bin/gather --instancetypes