This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [RFC] SystemTap future direction


On Thu, 2010-08-05 at 21:27 +0900, Masami Hiramatsu wrote:
> Mark Wielaard wrote:
> >>>> - Port SystemTap on the perf/ftrace but drop embedded-C support.
> >>>>  This will enhance perf/ftrace to support enough flexible data
> >>>>  filter/modifier (including fault injection feature). In this case,
> >>>>  SystemTap scripts will handle the data in user-space (not on-line).
> >>> I think the "not on-line" part is a bit of a showstopper. Since that
> >>> kills the main idea of having powerful scriptable observability. Simple
> >>> filters are too restrictive IMHO. It might be enough for simple
> >>> profiling, where you analyze the data off-line afterwards. But that
> >>> isn't an option for everybody (you need to store/push the data
> >>> somewhere), and not very efficient some cases.
> >> The efficiency is the key, and perf and systemtap aim to
> >> different efficiency. SystemTap focuses on the efficiency of
> >> transporting data, but perf focuses on the efficiency of
> >> probing time. What they are trying to is reducing the overhead
> >> of recording data to buffers, because it is less disturbance for
> >> the performance of target processes.
> > 
> > Right. It just comes down to priorities of the different goals.
> > Profiling (with offline analysis) versus scriptable tracing (with some
> > debugging elements). But making either efficient will help both cases.
> > We just have to be careful not to trade in one completely for the other,
> > or we kill useful use cases at probing time.
> 
> Hmm, could you find any useful use-case of scriptable tracing (on-line)?
> I know generally scriptable interface provides us great flexibility, but
> does it have to be done on-line? I mean, perf already has scriptable
> interface(python and perl) for off-line analysis. Why isn't that enough?

But you should know already! :) You wrote the kprobe-based event tracer!
Allowing to do online filtering based on simple register and memory
values on-line. So you don't need to do the filtering afterwards
off-line.

SystemTap just extends that basic idea with making on-line tracing
decision based on global variables, associate arrays, statistical
variables and some context gathering functions. All so you can easily
zoom into what is really important in your particular environment.

The off-line analysis through perf/python/perl is fine if you can afford
recording all the data you might be using at all the probe trace events.
And for some kinds of statistical profiling that is precisely what you
need anyway. Dump all data and analyse it off-line.

But this obviously has a cost. Say, you are interested in getting a
backtrace for when a certain event triggers, in the reference counting
gnome example when the count goes up to > 1024 for a particular object
type, but only if allocated from a particular module after startup,
which you mark as when the number of different objects created passes a
certain threshold. You can easily express that with systemtap using a
global variable and associate array. Only when the "context" is right do
you gather all the context data at the probe point, dump a stack trace
and the aggregated data. You can make that decision on-line before
gathering all the possibly relevant data at the probe point.

If you need to post-process off-line and don't have "on-line storage"
for keeping the statistics/associate arrays then you need to dump the
full context with every probe, including for example the backtraces
(which is expensive) since you don't know whether or not you might need
it. The post-processing off-line tools can of course then filter out all
unneeded data for you. But you still need to generate all the data
first, and then process it all (from a different context - context
switches aren't cheap either). That does disturb your system, especially
with larger data sets. If you are unlucky you could actually get into a
situation where the data dumping & post-processing tools generate more
i/o, syscalls, memory usage, etc. than the issue you are trying to
observe in the first place.

Cheers,

Mark


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]