[RFC] Crash extension for SystemTap

[Satoru MORIYA <satoru dot moriya dot br at hitachi dot com>]

Hi,

Here is an extension(shared object) of the crash to retrieve the trace
data of systemtap scripts.

I'd like to analyze what caused the kernel panic by using the systemtap.
However, currently the systemtap's trace data can't be retrieved from a
dumped image easily. So, I developed a crash's extension which retrieves
the data recorded by systemtap from the dumped image.
Here is a brief document of this extension. This extension supports the new
utt-based buffer as well as the bulk-mode buffer of old systemtap module.

I have tested this extention on the following system.
  * FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1
  * FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1
  * RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14


Preparation
==============
(A) Build the shared-object(stplog.so).

1. Put Makefile and stplog.c into a directory ($DIR)
    $ cd $DIR

2. Make the symbolic link to the crash source code directory
    $ ln -s $WHERE_CRASH_PLACED crash

3. Build
    $ make

(B) Make the crash dump which includes SystemTap trace data.
    (*)If you analyze the live system memory, ignore this section.

1. Install kdump
     If you use FC6, see following URL.
     http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29

2. Use SystemTap
    $ stap foo.stp

3. Panic
    $ echo c > /proc/sysrq-trigger

How to use
==============
1. start crash
    $ crash vmlinux vmcore
    (*) If you analyze the live system memory, you don't need "vmcore".
         $ crash vmlinux

2. load the shared-object
    crash> extend $(WHERE_OBJ_PLACED)/stplog.so

3. retrieve the data
    crash> stplog -m <mod_name>
    (*) <mod_name> is the name of trace module from which you retrieve data.

4. You can get output files under the directory whose name is <mod_name>.

Output
==============
stplog command makes a file per channel buffer of relayfs(equivalent to per cpu).
And it also removes padding bytes.


I believe this command is very useful for system administrators
if they monitor their systems with SystemTap.

Best Regards,

---
Satoru MORIYA
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: satoru.moriya.br@hitachi.com