Re: User-space probes: Plan B+

[Jim Keniston <jkenisto at us dot ibm dot com>]

On Fri, 2006-08-25 at 13:14, Frank Ch. Eigler wrote:
> jkenisto wrote:
> 
> > [...]
> > > > 1. Instrumentation can be coded entirely as a user-space app...
> > > 
> > > I am not aware of this being a *requirement*.  To my mind, it is an
> > > unfortunate implementation artifact.  Running in kernel space but in
> > > the traced processes' user context would be better.
> >
> > [...] Our interpretation of feedback from LKML and other sources was
> > that uprobes would not be accepted without a reasonably friendly
> > user-mode API.
> 
> That may be, but we would not be obligated to use such a thing.  To
> what extent can this work be layered on top of the kernel-side utrace
> API?

To a great extent, although we're still having to invent quite a bit. 
If you want SystemTap to generate kernel-only modules, then those
modules could probably use utrace directly except for the breakpoint
stuff.  But as the code matures, I'll see if providing a uprobes veneer
over utrace makes any sense for kernel instrumentation.  I doubt it
will.

> 
> 
> > A few things, such as stack backtraces with symbol info, cannot be
> > accomplished entirely in the kernel, so you need user-mode
> > cooperation at some point anyway.
> 
> That's sort of a separate issue.  We're not contemplating running live
> elfutils-style dwarf searches during a probe handler to evaluate
> backtraces.  It is more likely that information required for a precise
> backtrace would be encoded into the probing module in the form of an
> extract of the dwarf unwind tables, and searched in situ.

I get it.  Some apps are mighty big, though.  Do you envision encoding
enough dwarf info to do a gdb-style backtrace with arg values and
everything?  That could get huge.

> 
> 
> > > > [...]  3. A user-mode tracer can invoke a previously registered
> > > > kernel-mode handler, so we have simple and efficient communication
> > > > between user- and kernel-mode instrumentation. [...]
> > > 
> > > How is this registration aspect supposed to work?
> > 
> > Kernel-mode handlers can be registered in a kernel module. Registration
> > requires the handler address, a unique name, and permission info (e.g.,
> > only root can call it, or only a process that has permission to run
> > uprobe_register() on the traced process can call it).
> 
> > Handlers that we expect to have widespread use (e.g., collect
> > stack-trace addresses, report register contents, report memory
> > contents) could even be registered by the kernel.
> 
> FWIW, such designated "widespread-use" handlers are not in good match
> with systemtap's preference for programmable handlers.  It may be
> helpful for specialized purposes such as LKET (say, by some sort of
> clever handler body matching or new syntax), but not for general
> systemtap usage.

Got it.

> 
> > > Is something
> > > keeping us from compiling & registering this instrumentation
> > > dynamically, the same way we do normal kprobes handlers?
> >
> > No.  I envision SystemTap generating both the user-space tracer app
> > and the module that registers any needed ad hoc kernel handlers (as
> > well as the usual kernel-probing stuff).
> 
> Would this approach (compiling user-space probe handlers into
> kernel-space registered but custom-built utrace handlers) make the
> user-space tracer almost a content-free stub?

Again, if you want to go at it from the kernel side, we should restore
the old [un]register_uprobe kernel API, or something like it, as part of
Plan B+.  And again, you may want to use utrace directly to probe forks,
execs, signals, etc.

> 
> 
> - FChE

Jim