This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Discussion at Linux Foundation Japan Symposium


Hi -

On Mon, Dec 22, 2008 at 05:37:41PM -0500, Theodore Tso wrote:

> [...]
> > > Yes, but how long will people need to wait before debuginfo quality
> > > will improve?  And when such a hypothetical improved compiler be
> > > available and trusted to compile the kernel correctly?  [...]
> > 
> > You misinterpret the status quo.  In reality, debuginfo quality with
> > current gcc has been satisfactory for many purposes.  That is what
> > enables user-space debuggers to work as well as they do, and indeed
> > enables systemtap to work as well as it does.  The major current
> > debuginfo quality shortcoming involves only those intra-function
> > statement probes.
> 
> I can only speak from my personal experience, which is given the
> kernel default compilation options, and the inline functions and
> static functions in the parts of the kernel that I normally play with

Some inline/static functions become de facto intra-function statement
probes that we have problems with.  Note that can't probe those
functions based on symbol data (kallsyms) because they don't show up
there at all!  Same with ftrace, I think.  Keep that in mind if you
wish to fairly compare alternative technologies.


> [...]
> Worse yet, the error handling with Systemtap is so uniformly awful (to
> the point that even Roland has complained --- "stap -vvvvvvv" anyone?)

This has been since improved.


> It might be that the right answer is to tell kernel developers to
> compile their development kernels without optimization.  OK, it that
> goes against the Systemtap religion that you must never need to
> recompile or use special built options.  [...]

It's not a "religion".  It's sometimes an absolute user requirement.
It is also a way of letting the rest of the open-source "ecosystem"
evolve without us imposing ourselves.


> You can go ahead, and dismiss my reports [...]

I have endavoured to address each of your points.  If I overlooked
one, I apologize, and if you point it out I'll be glad to retry.


> > If you're still talking about debuginfo per se, I am led to believe
> > that only a small number of kernel developers are negatively affected:
> > those that build large kernel configurations but on such miniscule
> > hardware that the sheer disk I/O of the -g data is deemed intolerable.
> 
> Heh.  Only a very few number of kernel developers are even *trying* to
> use Systemtap at this point.  [...]

One short-term benefit you are likely to experience is the restoration
of the symbol-table-only probing, which should give you and others
some amount of probing capability without CONFIG_DEBUG_INFO.  Other
intermediate approaches (debuginfo subsetting, compression) appear to
require development well outside systemtap.

> After the Plumber's Conference and the Kernel Summit, [Arjan] van der
> Ven told me [...]  And other kernel engineers employed at Red Hat
> told me [...]

It is hard to respond to this, if these other gentlemen don't take up
our manifold invitations to comment/advise us directly as opposed to
each other.  (The last thing I recall Arjan ever suggesting was
packaging some systemtap tapset scripts along with the kernel, in the
hope that they would be maintained by subsystem maintainers there.
That idea appeared to be mooted by the apparent kernel summit
consensus for adding static compiled-in tracing hooks into the
kernel.)


> > > This means no modules, since debuginfo is a disaster with modules
> > > (so helping people create configurations that don't use modules
> > > would be helpful);
> > 
> > Please elaborate what you mean.  Why systemtap is forced to use
> > (create) modules at all has been explained often.

> What I mean is that if you build a kernel with a large number of
> modules, the bloat of systemtap becomes disastrous.  [...]

It's more fair to label this "bloat of debuginfo", since systemtap is
not responsible for how much code you choose to build.

> Unfortunately, building a distro-style kernel is a horrible disaster
> with debuginfo enabled, and so warning people off about this
> particular issue would be helpful.

Is http://tinyurl.com/96dza8 what you had in mind?

> > Sure, upstream changes in the location of Modules.marker are something
> > we can easily adapt to.  Are you aware of a current problem with that?
> > [...]
> 
> Only that "make install" or "make modules_install" doesn't currently
> install Modules.marker anywhere useful.  [...]

OK, I believe you mean that as the sole (?) user of this file,
systemtap folks should fix this kernel oversight by submitting a patch
against kbuild makefiles.


> > [...]  (As everyone knows, quantity (compression) is not great but
> > at least this is only a usability rather than functionality
> > matter.)
>
> It's a rather important usability issue.  [...]

Agreed.

> Line-by-line function tracing doesn't work for me.  [...]

I hope people will like it once it works robustly.

> User-space tracing --- is a patch that I don't forsee going mainline
> anytime soon [...]

I hope there will be progress on this soon.

> Concurrent users -- doesn't matter to kernel developers.  

I don't think that's completely true.  Right now, ftrace is a
single-configuration-at-a-time tool, and I recall reading complaints
about that.


> And exactly who do you expect is going to be adding markers to the
> kernel sources, [...]

It turns out that, among others, several Red Hat kernel engineers are
actively pursuing this, including sending tracepoint-laden patches
upstreamward via lkml/lttng.


> > > [...] is to simply generate the debuginfo information on demand so
> > > the function parameters on entry can be decoded.  We have the header
> > > files after all, so it shouldn't be hard to generate a pseudo function:
> > > 
> > > extern int foo_kernel_function(struct file *f, off_t offset, size_t len, 
> > >        void *ptr)
> > > {
> > >         return 0;
> > > }

> > It has some potential.  On first blush this seems to require
> > parsing all the C header files (accounting for arch/kconfig cpp
> > macros) to look for a matching function declaration.  [...]  How
> > do you imagine it working?

> Well, the original way I was thinking about doing this involved
> support from the kernel sources --- so changes to kbuild to generate
> debugging information that is far more compact, only optimized to
> provide kernel function parameter information

That's does not sound like a simple kbuild preference.  That may need
a whole new tool well beyond strip/objcopy.

> [...] However, it might be doable without having made any kernel
> changes.  Finding which header file a function parameter can be
> found is pretty simple.  [...]  The hard part is knowing which other
> header files might be needed.  This can be easily extracted from
> source files [...]

So more C source parsing.

> so the question is whether Systemtap ships a database of which
> header files are needed [...]

Maybe, but this sounds about as kernel-version-fragile as some of our
tapset scripts.

> or whether we need a kbuild patch to extract this information; or do
> both.

OK, so we agree we're talking about a nontrivial amount of code here.


- FChE


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]