$ oc get events -n <namespace>
OpenShift Virtualization provides tools and logs for troubleshooting virtual machines (VMs) and virtualization components.
You can troubleshoot OpenShift Virtualization components by using the tools provided in the web console or by using the oc
CLI tool.
Red Hat OpenShift Service on AWS events are records of important life-cycle information and are useful for monitoring and troubleshooting virtual machine, namespace, and resource issues.
VM events: Navigate to the Events tab of the VirtualMachine details page in the web console.
You can view namespace events by running the following command:
$ oc get events -n <namespace>
See the list of events for details about specific events.
You can view resource events by running the following command:
$ oc describe <resource> <resource_name>
You can view logs for OpenShift Virtualization pods by using the web console or the CLI. You can also view aggregated logs by using the LokiStack in the web console.
You can configure the verbosity level of OpenShift Virtualization pod logs by editing the HyperConverged
custom resource (CR).
To set log verbosity for specific components, open the HyperConverged
CR in your default text editor by running the following command:
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
Set the log level for one or more components by editing the spec.logVerbosityConfig
stanza. For example:
apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
spec:
logVerbosityConfig:
kubevirt:
virtAPI: 5 (1)
virtController: 4
virtHandler: 3
virtLauncher: 2
virtOperator: 6
1 | The log verbosity value must be an integer in the range 1–9 , where a higher number indicates a more detailed log. In this example, the virtAPI component logs are exposed if their priority level is 5 or higher. |
Apply your changes by saving and exiting the editor.
You can view the virt-launcher
pod logs for a virtual machine by using the Red Hat OpenShift Service on AWS web console.
Navigate to Virtualization → VirtualMachines.
Select a virtual machine to open the VirtualMachine details page.
On the General tile, click the pod name to open the Pod details page.
Click the Logs tab to view the logs.
You can view logs for the OpenShift Virtualization pods by using the oc
CLI tool.
View a list of pods in the OpenShift Virtualization namespace by running the following command:
$ oc get pods -n openshift-cnv
NAME READY STATUS RESTARTS AGE
disks-images-provider-7gqbc 1/1 Running 0 32m
disks-images-provider-vg4kx 1/1 Running 0 32m
virt-api-57fcc4497b-7qfmc 1/1 Running 0 31m
virt-api-57fcc4497b-tx9nc 1/1 Running 0 31m
virt-controller-76c784655f-7fp6m 1/1 Running 0 30m
virt-controller-76c784655f-f4pbd 1/1 Running 0 30m
virt-handler-2m86x 1/1 Running 0 30m
virt-handler-9qs6z 1/1 Running 0 30m
virt-operator-7ccfdbf65f-q5snk 1/1 Running 0 32m
virt-operator-7ccfdbf65f-vllz8 1/1 Running 0 32m
View the pod log by running the following command:
$ oc logs -n openshift-cnv <pod_name>
If a pod fails to start, you can use the To monitor log output in real time, use the |
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373695Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:453","timestamp":"2022-04-17T08:58:37.373726Z"}
{"component":"virt-handler","level":"info","msg":"setting rate limiter to 5 QPS and 10 Burst","pos":"virt-handler.go:462","timestamp":"2022-04-17T08:58:37.373782Z"}
{"component":"virt-handler","level":"info","msg":"CPU features of a minimum baseline CPU model: map[apic:true clflush:true cmov:true cx16:true cx8:true de:true fpu:true fxsr:true lahf_lm:true lm:true mca:true mce:true mmx:true msr:true mtrr:true nx:true pae:true pat:true pge:true pni:true pse:true pse36:true sep:true sse:true sse2:true sse4.1:true ssse3:true syscall:true tsc:true]","pos":"cpu_plugin.go:96","timestamp":"2022-04-17T08:58:37.390221Z"}
{"component":"virt-handler","level":"warning","msg":"host model mode is expected to contain only one model","pos":"cpu_plugin.go:103","timestamp":"2022-04-17T08:58:37.390263Z"}
{"component":"virt-handler","level":"info","msg":"node-labeller is running","pos":"node_labeller.go:94","timestamp":"2022-04-17T08:58:37.391011Z"}
Viewing the boot logs of VM guests can help diagnose issues. You can configure access to guests' logs and view them by using either the Red Hat OpenShift Service on AWS web console or the oc
CLI.
This feature is disabled by default. If a VM does not explicitly have this setting enabled or disabled, it inherits the cluster-wide default setting.
If sensitive information such as credentials or other personally identifiable information (PII) is written to the serial console, it is logged with all other visible text. Red Hat recommends using SSH to send sensitive data instead of the serial console. |
You can enable default access to VM guest system logs by using the web console.
From the side menu, click Virtualization → Overview.
Click the Settings tab.
Click Cluster → Guest management.
Set Enable guest system log access to on.
You can enable default access to VM guest system logs by editing the HyperConverged
custom resource (CR).
Open the HyperConverged
CR in your default editor by running the following command:
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
Update the disableSerialConsoleLog
value. For example:
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
spec:
virtualMachineOptions:
disableSerialConsoleLog: true (1)
#...
1 | Set the value of disableSerialConsoleLog to false if you want serial console access to be enabled on VMs by default. |
You can configure access to VM guest system logs for a single VM by using the web console. This setting takes precedence over the cluster-wide default configuration.
Click Virtualization → VirtualMachines from the side menu.
Select a virtual machine to open the VirtualMachine details page.
Click the Configuration tab.
Set Guest system log access to on or off.
You can configure access to VM guest system logs for a single VM by editing the VirtualMachine
CR. This setting takes precedence over the cluster-wide default configuration.
Edit the virtual machine manifest by running the following command:
$ oc edit vm <vm_name>
Update the value of the logSerialConsole
field. For example:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: example-vm
spec:
template:
spec:
domain:
devices:
logSerialConsole: true (1)
#...
1 | To enable access to the guest’s serial console log, set the logSerialConsole value to true . |
Apply the new configuration to the VM by running the following command:
$ oc apply vm <vm_name>
Optional: If you edited a running VM, restart the VM to apply the new configuration. For example:
$ virtctl restart <vm_name> -n <namespace>
You can view the serial console logs of a virtual machine (VM) guest by using the web console.
Guest system log access is enabled.
Click Virtualization → VirtualMachines from the side menu.
Select a virtual machine to open the VirtualMachine details page.
Click the Diagnostics tab.
Click Guest system logs to load the serial console.
You can view the serial console logs of a VM guest by running the oc logs
command.
Guest system log access is enabled.
View the logs by running the following command, substituting your own values for <namespace>
and <vm_name>
:
$ oc logs -n <namespace> -l kubevirt.io/domain=<vm_name> --tail=-1 -c guest-console-log
You can facilitate troubleshooting by aggregating and filtering logs.
You can view aggregated logs for OpenShift Virtualization pods and containers by using the LokiStack in the web console.
You deployed the LokiStack.
Navigate to Observe → Logs in the web console.
Select application, for virt-launcher
pod logs, or infrastructure, for OpenShift Virtualization control plane pods and containers, from the log type list.
Click Show Query to display the query field.
Enter the LogQL query in the query field and click Run Query to display the filtered logs.
You can view and filter aggregated logs for OpenShift Virtualization components by running Loki Query Language (LogQL) queries on the Observe → Logs page in the web console.
The default log type is infrastructure. The virt-launcher
log type is application.
Optional: You can include or exclude strings or regular expressions by using line filter expressions.
If the query matches a large number of logs, the query might time out. |
Component | LogQL query | ||
---|---|---|---|
All |
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
Container |
|
||
|
You must select application from the log type list before running this query.
|
You can filter log lines to include or exclude strings or regular expressions by using line filter expressions.
Line filter expression | Description |
---|---|
|
Log line contains string |
|
Log line does not contain string |
|
Log line contains regular expression |
|
Log line does not contain regular expression |
{log_type=~".+"}|json
|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
|= "error" != "timeout"
xref :../../observability/logging/log_storage/about-log-storage.adoc#about-log-storage[About log storage]
LogQL log queries in the Grafana documentation
The following error messages might appear in OpenShift Virtualization logs:
ErrImagePull
or ImagePullBackOff
Indicates an incorrect deployment configuration or problems with the images that are referenced.
You can check the Conditions
and Events
sections of the DataVolume
object to analyze and resolve issues.
You can diagnose data volume issues by examining the output of the Conditions
and Events
sections
generated by the command:
$ oc describe dv <DataVolume>
The Conditions
section displays the following Types
:
Bound
Running
Ready
The Events
section provides the following additional information:
Type
of event
Reason
for logging
Source
of the event
Message
containing additional diagnostic information.
The output from oc describe
does not always contains Events
.
An event is generated when the Status
, Reason
, or Message
changes.
Both conditions and events react to changes in the state of the data volume.
For example, if you misspell the URL during an import operation, the import
generates a 404 message. That message change generates an event with a reason.
The output in the Conditions
section is updated as well.
By inspecting the Conditions
and Events
sections generated by the describe
command, you determine the state of the data volume
in relation to persistent volume claims (PVCs), and whether or
not an operation is actively running or completed. You might also receive messages
that offer specific details about the status of the data volume, and how
it came to be in its current state.
There are many different combinations of conditions. Each must be evaluated in its unique context.
Examples of various combinations follow.
Bound
- A successfully bound PVC displays in this example.
Note that the Type
is Bound
, so the Status
is True
.
If the PVC is not bound, the Status
is False
.
When the PVC is bound, an event is generated stating that the PVC is bound.
In this case, the Reason
is Bound
and Status
is True
.
The Message
indicates which PVC owns the data volume.
Message
, in the Events
section, provides further details including how
long the PVC has been bound (Age
) and by what resource (From
),
in this case datavolume-controller
:
Status:
Conditions:
Last Heart Beat Time: 2020-07-15T03:58:24Z
Last Transition Time: 2020-07-15T03:58:24Z
Message: PVC win10-rootdisk Bound
Reason: Bound
Status: True
Type: Bound
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Bound 24s datavolume-controller PVC example-dv Bound
Running
- In this case, note that Type
is Running
and Status
is False
,
indicating that an event has occurred that caused an attempted
operation to fail, changing the Status from True
to False
.
However, note that Reason
is Completed
and the Message
field indicates
Import Complete
.
In the Events
section, the Reason
and Message
contain additional
troubleshooting information about the failed operation. In this example,
the Message
displays an inability to connect due to a 404
, listed in the
Events
section’s first Warning
.
From this information, you conclude that an import operation was running, creating contention for other operations that are attempting to access the data volume:
Status:
Conditions:
Last Heart Beat Time: 2020-07-15T04:31:39Z
Last Transition Time: 2020-07-15T04:31:39Z
Message: Import Complete
Reason: Completed
Status: False
Type: Running
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Error 12s (x2 over 14s) datavolume-controller Unable to connect
to http data source: expected status code 200, got 404. Status: 404 Not Found
Ready
– If Type
is Ready
and Status
is True
, then the data volume is ready
to be used, as in the following example. If the data volume is not ready to be
used, the Status
is False
:
Status:
Conditions:
Last Heart Beat Time: 2020-07-15T04:31:39Z
Last Transition Time: 2020-07-15T04:31:39Z
Status: True
Type: Ready