To troubleshoot some issues with nodes, establish the kubelet’s log level verbosity depending on the issue to be tracked.

Modifying the kubelet as a one-time scenario

To modify the kubelet in a one-time scenario without rebooting the node due to the change of machine-config(spec":{"paused":false}}), allowing you to modify the kubelet without affecting the service, follow this procedure.

Procedure
  1. Connect to the node in debug mode:

    $ oc debug node/<node>
    $ chroot /host
  2. After access is established, check the content:

    $ systemctl cat kubelet
    Example output
    # /etc/systemd/system/kubelet.service
    mode: 0644
    path: "/etc/systemd/system/kubelet.service.d/20-logging.conf"
    contents:
      inline: |
        [Service]
        Environment="KUBELET_LOG_LEVEL=2"
  3. Define the new verbosity required in the /etc/systemd/system/kubelet.service.d/20-logging.conf file. In this example, the verbosity is changed from v=1 to v=8:

    $ vi -i -e 's/--v=1/--v=8/g' /etc/systemd/system/kubelet.service.d/20-logging.conf

    Editing the config file or installing a new logging.conf file overrides the log level.

  4. Restart the service:

    $ systemctl daemon-reload
    $ systemctl restart kubelet
  5. Gather the logs, then edit the kubelet log level to revert to the former value to prevent issues, such as this error:

    E0514 12:47:17.998892    2281 daemon.go:1350] content mismatch for file /etc/systemd/system/kubelet.service: [Unit]

Persistent kubelet log level configuration

Procedure
  • Use the following MachineConfig object for persistent kubelet log level configuration:

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
     metadata:
       labels:
         machineconfiguration.openshift.io/role: master
       name: 99-master-kubelet-loglevel
     spec:
       config:
         ignition:
           version: 3.2.0
         systemd:
           units:
             - name: kubelet.service
               enabled: true
               dropins:
                 - name: 30-logging.conf
                   contents: |
                     [Service]
                     Environment="KUBELET_LOG_LEVEL=2"

    Generally, it is recommended to apply 0-4 as debug-level logs and 5-8 as trace-level logs.

Log verbosity descriptions

Log verbosity Description

--v=0

Always visible to an Operator.

--v=1

A reasonable default log level if you do not want verbosity.

--v=2

Useful steady state information about the service and important log messages that might correlate to significant changes in the system. This is the recommended default log level.

--v=3

Extended information about changes.

--v=4

Debug level verbosity.

--v=6

Display requested resources.

--v=7

Display HTTP request headers.

--v=8

Display HTTP request contents.

Gathering kubelet logs

Procedure
  • After the kubelet’s log level verbosity is configured properly, you can gather logs by running the following commands:

    $ oc adm node-logs --role master -u kubelet
    $ oc adm node-logs --role worker -u kubelet

    Alternatively, inside the node, run the following command:

    $ journalctl -b -f -u kubelet.service
  • To collect master container logs, run the following command:

    $ sudo tail -f /var/log/containers/*
  • To directly gather the logs of all nodes, run the following command:

    - for n in $(oc get node --no-headers | awk '{print $1}'); do oc adm node-logs $n | gzip > $n.log.gz; done