# rpm-ostree kargs --append='crashkernel=256M'
OpenShift Container Platform runs on RHCOS. You can follow these procedures to troubleshoot problems related to the operating system.
The kdump
service, included in kexec-tools
, provides a crash-dumping mechanism. You can use this service to save the contents of the system’s memory for later analysis.
The kdump
service is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
RHCOS ships with kexec-tools
, but manual configuration is required to enable kdump
.
Perform the following steps to enable kdump
on RHCOS.
To reserve memory for the crash kernel during the first kernel booting, provide kernel arguments by entering the following command:
# rpm-ostree kargs --append='crashkernel=256M'
By default, the path in which the vmcore will be saved is /var/crash
. It is also possible to write the dump over the network or to some other location on the local system by editing /etc/kdump.conf
. For example, assuming /var/usrlocal/cores
exists, enter the following command to edit /etc/kdump.conf
to save the vmcore to /var/usrlocal/cores
:
# sed -i "s/^path.*/path \/var\/usrlocal\/cores/" /etc/kdump.conf
For additional information, see kdump.conf
, a manual page for the /etc/kdump.conf
configuration file containing the full documentation of available options, and note the comments in /etc/kdump.conf
and /etc/sysconfig/kdump
.
Also refer to the RHEL kdump
documentation for further information on configuring the dump target.
Because kdump
has trouble finding the correct bootimage location on RHCOS, the KDUMP_BOOTDIR
variable must be manually set in /etc/kdump.conf
. You can use /proc/cmdline
to figure out the ostree
boot location. For example:
# BOOT_LOC=/boot$(cat /proc/cmdline | egrep -o "/ostree/.*/vmlinuz" | sed -e "s|/vmlinuz||g")
# sed -i "s|^#KDUMP_BOOTDIR=\"/boot\"|KDUMP_BOOTDIR=\"${BOOT_LOC}\"|" /etc/sysconfig/kdump
Configure /etc/sysconfig/kdump
to not use the default file-based kexec
syscall for loading the crash kernel.
# sed -i "s|^KEXEC_ARGS=\"-s\"|KEXEC_ARGS=\"\"|" /etc/sysconfig/kdump
Enable the kdump
systemd service.
# systemctl enable kdump.service
Reboot your system.
# systemctl reboot
Ensure that kdump
has loaded a crash kernel by checking that the kdump.service
has started and exited successfully and that cat /sys/kernel/kexec_crash_loaded
prints 1
.
The kdump
service is intended to be enabled per node to debug kernel problems. Because there are costs to having kdump
enabled, and these costs accumulate with each additional kdump
-enabled node, it is recommended that kdump
only be enabled on each node as needed. Potential costs of enabling kdump
on each node include:
Less available RAM due to memory being reserved for the crash kernel.
Node unavailability while the kernel is dumping the core.
Additional storage space being used to store the crash dumps.
Not being production-ready because the kdump
service is in Technology Preview.
If you are aware of the downsides and trade-offs of having the kdump
service enabled, it is possible to enable kdump
in a cluster-wide fashion. Although machine-specific machine configs are not yet supported, you can perform the previous steps through a systemd
unit in a MachineConfig
object on day-1 and have kdump enabled on all nodes in the cluster. You can create a MachineConfig
object and inject that object into the set of manifest files used by Ignition during cluster setup. See "Customizing nodes" in the Installing → Installation configuration section for more information and examples on how to use Ignition configs.
See the Testing the kdump configuration section in the RHEL documentation for kdump
.
See the Analyzing a core dump section in the RHEL documentation for kdump
.
kdump.conf(5) — a manual page for the /etc/kdump.conf
configuration file containing the full documentation of available options
kexec(8) — a manual page for kexec
Red Hat Knowledgebase article regarding kexec
and kdump
.