This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: [PATCH tracing/kprobes 0/7] tracing/kprobes: kprobe-based event tracer update and perf support
- From: Frederic Weisbecker <fweisbec at gmail dot com>
- To: Masami Hiramatsu <mhiramat at redhat dot com>
- Cc: Steven Rostedt <rostedt at goodmis dot org>, Ingo Molnar <mingo at elte dot hu>, lkml <linux-kernel at vger dot kernel dot org>, Ananth N Mavinakayanahalli <ananth at in dot ibm dot com>, Andi Kleen <ak at linux dot intel dot com>, Christoph Hellwig <hch at infradead dot org>, "Frank Ch. Eigler" <fche at redhat dot com>, "H. Peter Anvin" <hpa at zytor dot com>, Jason Baron <jbaron at redhat dot com>, Jim Keniston <jkenisto at us dot ibm dot com>, "K.Prasad" <prasad at linux dot vnet dot ibm dot com>, Lai Jiangshan <laijs at cn dot fujitsu dot com>, Li Zefan <lizf at cn dot fujitsu dot com>, Peter Zijlstra <peterz at infradead dot org>, Srikar Dronamraju <srikar at linux dot vnet dot ibm dot com>, Tom Zanussi <tzanussi at gmail dot com>, systemtap <systemtap at sources dot redhat dot com>, DLE <dle-develop at lists dot sourceforge dot net>
- Date: Fri, 11 Sep 2009 03:33:33 +0200
- Subject: Re: [PATCH tracing/kprobes 0/7] tracing/kprobes: kprobe-based event tracer update and perf support
- References: <20090910235258.22412.29317.stgit@dhcp-100-2-132.bos.redhat.com>
On Thu, Sep 10, 2009 at 07:52:58PM -0400, Masami Hiramatsu wrote:
> Hi Frederic,
>
> This series fixes bugs and upgrades kprobe-based event tracer
> as a dynamic event tracer on ftrace/perf tools. This also enhances
> tracer output format to show each argument name and event name on
> each entry.
>
> With this series, users can add trace events dynamically on ftrace
> and use those events with perf tools as below.
>
> (Step.1) Define new events under new group
>
> $ echo p:mygroup/myprobe do_sys_open dfd=a0 filename=a1 flags=a2 mode=a3 \
> > /debug/tracing/kprobes_events
> $ echo r:mygroup/myretprobe do_sys_open rv >> /debug/tracing/kprobes_events
> $ cat /debug/tracing/kprobes_events
> p:myprobe do_sys_open+0 dfd=a0 filename=a1 flags=a2 mode=a3
> r:myretprobe do_sys_open+0 rv=rv
>
> (You can see that each argument has its name.)
>
>
> (Step.2) Perf shows new events
>
> $ perf list
> ...
> mygroup:myretprobe [Tracepoint event]
> mygroup:myprobe [Tracepoint event]
> ...
>
>
> (Step.3) Record events with perf
>
> $ perf record -f -e mygroup:myprobe:record -F 1 -a ls
> ...
> [ perf record: Captured and wrote 0.081 MB perf.data (~3544 samples) ]
>
>
> (Step.4) Perf trace shows the result
>
> $ perf trace
> version = 0.5
> perf-1405 [000] 0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=810d3f7 flags=98800 mode=1
> perf-1405 [000] 0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=bff7650c flags=8000 mode=1b6
> perf-1405 [000] 0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=bff7650c flags=98800 mode=bff7450c
> perf-1405 [000] 0.000000: myprobe: (c04b0a5c) dfd=ffffff9c filename=bff7650c flags=8000 mode=1b6
Nice!
May be another step in the todo-list that would be nice: define the format
for a type. Like it's done from ftrace events.
>
>
> (Step.5) You can also use return probes.
>
> $ perf record -f -e mygroup:myretprobe:record -F 1 -a ls
> ...
> [ perf record: Captured and wrote 0.081 MB perf.data (~3544 samples) ]
> $ perf trace
> version = 0.5
> perf-1408 [000] 0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=b
> perf-1408 [000] 0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=c
> perf-1408 [000] 0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=d
> perf-1408 [000] 0.000000: myretprobe: (c04b0a5c <- c04b0b7d) rv=c
>
>
> TODO:
> - Implement perf kprobe command to help defining new probes.
Yeah!
I wonder what could be the best workflow to use it.
Imagine the following steps:
- perf kprobe = define kprobes using C expression
- perf record -e our_kprobes
- perf trace
That's way too much.
Especially it's sad to be forced to define a kprobe, then
get back its name, use it with record, and eventually
unsheathe perf trace.
I guess we should choose between the low level, very granular
but uninviting method "kprobe + record + trace" and also an all
in one quick approach.
And that could be chosen from perf kprobe:
Low level:
perf kprobe --define-only [-p|-r] [probe_name] -a1 [arg1] -a2 [arg2] \
--format="%s %...."
perf record -e kprobes:probe_name
perf trace
Quick:
perf kprobe -p probe_name -a1 ..... cmdline| -a
And after the profiled task is finished, it could launch perf trace
by itself (or wait for a Ctrl + C if -a/wide profiling)
Well, it's just a brainstorming, having the low level method first
would be already a very nice thing.
I'm really looking forward seeing this C expression-like kprobe creation
tool.
It seems powerful enough to replace printk + kernel rebuild.
No need anymore to write some printk to debug, worrying,
sweating, feeling guilty because we know we'll need yet another
printk() after the reboot, and we even already know where while
it is compiling.
We would build less kernels, then drink less coffee, becoming
less nervous, more friendly. Everyone will offer flowers in
the street, the icebergs will grow back and white bears will...
And eventually we'll be inspired enough to write perf love,
the more than expected tool to post process ftrace "love" events.
Thanks,
Frederic.