How to debug crashing application on RHEL/Centos systems

We are running bunch of our applications on production Centos 5 servers and it happens sometimes that one of the processes crashes unexpectedly but not so often so that we could reproduce the behaviour and localize the problem. At this point having a possibility to turn on the debug mode in Linux definitely comes in handy.

These are simple things that could help you to find out why the program crashes:

1. Enable logging userspace segfaults to the kernel console
   echo 1 > /proc/sys/kernel/print-fatal-signals
   To make this change permanent also after reboot, insert this line into /etc/sysctl.conf:
   kernel.print-fatal-signals = 1 
2. Enable core dumps before you run the application from the shell
   ulimit -c unlimitedFor those applications started from init scripts within /etc/init.d (and using RHEL /etc/rc.d/init.d/functions) you can
   1. globally enable core dump by inserting this line into /etc/sysconfig/init:
      DAEMON_COREFILE_LIMIT=’unlimited’
   2. enable core dumps for specific daemon only by editing respective init script and inserting DAEMON_COREFILE_LIMIT=’unlimited’ after “. /etc/rc.d/init.d/functions” line.

   More information about what can be done with ulimit can be found here.
   Here is detailed information about other possibilities how to enable core dumps.

Posted in Server administration