streaming output options (was Re: [external-perftools-list] notes from 20070503 Systemtap Meeting)

[Martin Hunt <hunt at redhat dot com>]

On Tue, 2007-05-08 at 13:26 -0400, Frank Ch. Eigler wrote:

> I believe the primary concern should not be the overall speed of
> merging, but rather, the non-intrusiveness of the trace writing
> process.  Specifically, two processors writing trace data at the same
> time should not casually block each other.

I thought your concern was with having a single easy-to-read output
channel?

Let me explain in more detail.  Perhaps I am missing some options, but
these seem to be the options we have.

1. A single file that contains the data.  Can use with "cat",
third-party apps. To do this, merging must be in the kernel. That means
some blocking must occur.  On my test system, I'm showing that with two
processors in a tight loop writing to a single output channel, a probe
that does a 100 byte printf takes 350ns in bulk mode compared to 400ns
in streaming mode where it must get spinlock then memcpy the data to the
channel. This is worst case. Note that 100bytes every 350ns = 272
Mbytes/sec, which is a very fast logging rate.

2. Per-cpu files that are merged into a single stream in userspace. This
stream can be made available to other apps, but that merging will be
tricky and require that the apps either use staprun or some library we
provide (UTT?). This option is not flight-recorder friendly because
unsent output will be spread over the percpu files and cannot be simple
ASCII (it must have headers sequence numbers and lengths). Perhaps
someone more experience with kdump can suggest some way to make the data
more accessible?

Anything else?

Martin