Configure kdump to analyse Linux kernel crash and kernel panics.
Linux kernel crash and panics and the reason behind it is analyzed with the help of kdump utility.
Kdump is a crash dumping mechanism and it uses the context of another kernel at boot time to capture the crash and core dump. The context of the kernel reserves a small amount of memory, and its only purpose is to capture the core dump of the crashed kernel.
Following are the steps to configure kdump from command prompt.
Login as a user root and edit /boot/grub/grub.conf file, and add the crashkernel=M parameter to the list of kernel options. After editing grub.conf file looks like as follows:
# grub.conf generated by anaconda
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-version.img
title Red Hat Enterprise Linux Server (2.6.18-194.8.1.el5)
kernel /vmlinuz-2.6.18-194.8.1.el5 ro root=/dev/sda3 crashkernel=128M
Now configure the target location in /etc/kdump.conf file. The default file location to store dump file i.e. vmcore files in the /var/crash/ directory of the local system.
It is also possible to save the file on remote location using NFS and SCP, but these techniques are not written here.
Edit /etc/kdump.conf file:
To change the local directory in which the core dump is to be saved, remove the hash sign (#) from the beginning of the #path /var/crash line, and replace the value with a desired directory path.
To write the dump directly to a device, remove the hash sign (#) from the beginning of the #raw /dev/sdc5 line, and replace the value with a desired device name. For example:
Also possible to configure dump core using core collector. To reduce the size of the vmcore we can use makedumpfile utility.
To enable core collector search for core_collector directive in /etc/kdump.conf file and uncomment it if it is not.
core_collector makedumpfile c
To remove the unwanted pages from the dump file we can pass d option to core_collector.
core_collector makedumpfile -d c
where Numeric value is a sum of values of pages we want to omit.
Option Page type to omit
1 Zero Pages
2 Cache Pages
4 Cache Private
8 User Pages
16 Blank Pages
Now all we need is to start the kdump service on boot time.
#chkconfig kdump on
Start the kdump service.
# service kdump start
No kdump initial ramdisk found. [WARNING]
Starting kdump: [ OK ]
Test the kdump configuration.
To test the configuration, reboot the system with kdump enabled, and make sure that the service is running:
# service kdump status
Kdump is operational
Then type the following commands at a shell prompt:
# echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger
The above command makes Linux kernel to crash, and the YYYY-MM-DD-HH:MM/vmcore file will be copied to the location we have selected in the configuration.
Analyze the core dump:
To analyze the crash kernel we need to packages and they are crash and kernel-debuginfo.
Now all we need to start the crash utility.
crash /var/crash/timestamp/vmcore /usr/lib/debug/lib/modules/kernel/vmlinux.
At the crash prompt we can run several commands like
Crash> log ## Display the mesg buffer
Crash>bt ## backtrace
Crash>[ps | vm | files] ## Refer man page for more options.