Collecting data for Red Hat Support - Support | Virtualization

Collecting data about your environment
Collecting data about virtual machines
Using the must-gather tool for OpenShift Virtualization
- must-gather tool options

When you submit a support case to Red Hat Support, it is helpful to provide debugging information for OpenShift Container Platform and OpenShift Virtualization by using the following tools:

must-gather tool: The must-gather tool collects diagnostic information, including resource definitions and service logs.
Prometheus: Prometheus is a time-series database and a rule evaluation engine for metrics. Prometheus sends alerts to Alertmanager for processing.
Alertmanager: The Alertmanager service handles alerts received from Prometheus. The Alertmanager is also responsible for sending the alerts to external notification systems.

For information about the OpenShift Container Platform monitoring stack, see About OpenShift Container Platform monitoring.

Collecting data about your environment

Collecting data about your environment minimizes the time required to analyze and determine the root cause.

Prerequisites

Set the retention time for Prometheus metrics data to a minimum of seven days.
Configure the Alertmanager to capture relevant alerts and to send alert notifications to a dedicated mailbox so that they can be viewed and persisted outside the cluster.
Record the exact number of affected nodes and virtual machines.

Procedure

Collecting data about virtual machines

Collecting data about malfunctioning virtual machines (VMs) minimizes the time required to analyze and determine the root cause.

Prerequisites

Linux VMs: Install the latest QEMU guest agent.
Windows VMs:
- Record the Windows patch update details.
- Install the latest VirtIO drivers.
- Install the latest QEMU guest agent.
- If Remote Desktop Protocol (RDP) is enabled, connect by using the desktop viewer to determine whether there is a problem with the connection software.

Procedure

Collect must-gather data for the VMs using the /usr/bin/gather script.
Collect screenshots of VMs that have crashed before you restart them.
Collect memory dumps from VMs before remediation attempts.
Record factors that the malfunctioning VMs have in common. For example, the VMs have the same host or network.

Using the must-gather tool for OpenShift Virtualization

You can collect data about OpenShift Virtualization resources by running the must-gather command with the OpenShift Virtualization image.

The default data collection includes information about the following resources:

OpenShift Virtualization Operator namespaces, including child objects
OpenShift Virtualization custom resource definitions
Namespaces that contain virtual machines
Basic virtual machine definitions

Instance types information is not currently collected by default; you can, however, run a command to optionally collect it.

Procedure

Run the following command to collect data about OpenShift Virtualization:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.17.2 \
  -- /usr/bin/gather

must-gather tool options

You can run the oc adm must-gather command to collect must gather images for all the Operators and products deployed on your cluster without the need to explicitly specify the required images. Alternatively, you can specify a combination of scripts and environment variables for the following options:

Collecting detailed virtual machine (VM) information from a namespace
Collecting detailed information about specified VMs
Collecting image, image-stream, and image-stream-tags information
Limiting the maximum number of parallel processes used by the must-gather tool

Parameters

Environment variables

You can specify environment variables for a compatible script.

NS=<namespace_name>

Collect virtual machine information, including virt-launcher pod details, from the namespace that you specify. The VirtualMachine and VirtualMachineInstance CR data is collected for all namespaces.

VM=<vm_name>

Collect details about a particular virtual machine. To use this option, you must also specify a namespace by using the NS environment variable.

PROS=<number_of_processes>

Modify the maximum number of parallel processes that the must-gather tool uses. The default value is 5.

Using too many parallel processes can cause performance issues. Increasing the maximum number of parallel processes is not recommended.

Scripts

Each script is compatible only with certain environment variable combinations.

/usr/bin/gather: Use the default must-gather script, which collects cluster data from all namespaces and includes only basic VM information. This script is compatible only with the PROS variable.
/usr/bin/gather --vms_details: Collect VM log files, VM definitions, control-plane logs, and namespaces that belong to OpenShift Virtualization resources. Specifying namespaces includes their child objects. If you use this parameter without specifying a namespace or VM, the must-gather tool collects this data for all VMs in the cluster. This script is compatible with all environment variables, but you must specify a namespace if you use the VM variable.
/usr/bin/gather --images: Collect image, image-stream, and image-stream-tags custom resource information. This script is compatible only with the PROS variable.
/usr/bin/gather --instancetypes: Collect instance types information. This information is not currently collected by default; you can, however, optionally collect it.

Usage and examples

Environment variables are optional. You can run a script by itself or with one or more compatible environment variables.

Table 1. Compatible parameters
Script	Compatible environment variable
`/usr/bin/gather`	* `PROS=<number_of_processes>`
`/usr/bin/gather --vms_details`	* For a namespace: `NS=<namespace_name>` * For a VM: `VM=<vm_name> NS=<namespace_name>` * `PROS=<number_of_processes>`
`/usr/bin/gather --images`	* `PROS=<number_of_processes>`

Syntax

To collect must-gather logs for all Operators and products on your cluster in a single pass, run the following command:

$ oc adm must-gather --all-images

If you need to pass additional parameters to individual must-gather images, use the following command:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.17.2 \
  -- <environment_variable_1> <environment_variable_2> <script_name>

Default data collection parallel processes

By default, five processes run in parallel.

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.17.2 \
  -- PROS=5 /usr/bin/gather (1)

1	You can modify the number of parallel processes by changing the default.

Detailed VM information

The following command collects detailed VM information for the my-vm VM in the mynamespace namespace:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.17.2 \
  -- NS=mynamespace VM=my-vm /usr/bin/gather --vms_details (1)

1	The `NS` environment variable is mandatory if you use the `VM` environment variable.

Image, image-stream, and image-stream-tags information

The following command collects image, image-stream, and image-stream-tags information from the cluster:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.17.2 \
  /usr/bin/gather --images

Instance types information

The following command collects instance types information from the cluster:

$ oc adm must-gather \
  --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.17.2 \
  /usr/bin/gather --instancetypes