Re: DTrace on BSD, future support

[Mark Wielaard <mark at klomp dot org>]

Hi,

Adding the systemtap list which has some smart people on it who
can probably help out.

Disclaimer: I have played a bit and hacked a bit on Systemtap
and as Andrew said I did make dtrace probes in hotspot work
nicely together with Systemtap a couple of years ago:
http://gnu.wildebeest.org/blog/mjw/2009/08/06/i-love-it-when-a-plan-comes-together-systemtap-meets-java/
But I never have used macosx and the few times I tried to use
Solaris/OpenIndiana dtrace it crashed my system. So I don't
really know much about dtrace itself. When I added the dtrace
SDT probe support on GNU/Linux I did ask for the hotspot/dtrace
testcases to make sure everything was working well. But these
testcases are apparently secret making it hard to make any
compatibility claims :{

On Fri, Sep 06, 2013 at 11:22:42PM -0400, Andrew Hughes wrote:
> > I'm working with a GSoC student adding dtrace probe support to JRuby,
> > and ran into the issue that user-defined dtrace probes only work on
> > Solaris. So I started poking around hotspot source and found that both
> > the linux and bsd JSDT sources are empty and return false for
> > isSupported. Only solaris has JSDT support.
> > 
> Linux doesn't use DTrace.  It uses SystemTap.
> 
> CCing Mark Wielaard who added this support:
> 
> http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/rev/c707a5af0d71

"added support" is a bit much for that simple hack. The support
was always there, all I did was tweak the Makefile a bit to get
it compiled in (yes, that simple patch took months to get accepted,
but that wasn't because the patch was deep or difficult).

On GNU/Linux sys/sdt.h probes are available to systemtap, gdb and perf
simply by recompiling the source with a recent enough GCC. The DTRACE
markers are source compatible, so with that simple tweak you can now
easily use them from any of the observability, debugging, tracing or
profiling tools on GNU/Linux that can read SDT probes in ELF files.
That is in general the case with any dtrace SDT probes. Just rebuild
the sources with /usr/include/sys/sdt.h installed (and maybe a
small configure check or define added) and tada!

> > So at this point I'm wondering a few things:
> > 
> > * Is there any good reason why BSD (and Linux) support for JSDT has
> > not been ported or implemented?
> > * What is the current thought on the future of DTrace support in OpenJDK?
> > 
> > Our steps going forward, if we want to support JRuby dtrace probes on
> > platforms other than Solaris, will be to either implement or coerce
> > someone into implementing BSD/Linux DTrace JSDT support in OpenJDK or
> > to use a third-party library and our own JNI bindings. I'd prefer to
> > just get it working in OpenJDK proper.
> > 
> > Thoughts?

JSDT is a general framework for providing some tracing support in
hotspot/openjdk introduced with 1.7. One provider is based on dtrace
(solaris specific atm). I am not sure there is "official" documentation
for it. It openjdk specific and lives in the private package
com.sun.tracing.*.

One idea is to just implement a tracing provider through Byteman.
https://www.jboss.org/byteman That would give you a "pure java"
tracing factory, which might interface easier with JRuby.
As a bonus systemtap comes with some Byteman hooks to use Byteman
rewrite rules to make trace points visible as SDT probes in a
JNI library that can be observed by systemtap.
https://lukasberk.wordpress.com/2013/06/20/probing-java-methods-with-systemtap/

The JSDT DTraceProviderFactory makes use of a feature that is dtrace/kernel
specific. So it takes a bit more to implement it than just a recompile
as is the case with normal SDT markers. It opens a special file
/dev/dtrace/helper with which it does a secret handshake to tell dtrace
on solaris where the hotspot JIT created markers in the code and the
name it wants these probes points to have. This is communicated by
creating a fake SUNW_DOF ELF section that is specific to how dtrace
handles SDT probes.

There doesn't seem to be much documentation on these dtrace specific
special file helper and the SUNW_DOF ELF section. But the concept is
clear. For GNU/Linux we currently don't have such a registration service
for new SDT probes. So we should create one. Probably based on the
stapsdt NOTEs that the various tools already use to detect SDT probes.
I am not sure it needs to be a special linux kernel device or proc file.
Maybe it can be a generic daemon (using dbus to register and query
which probe providers are currently active maybe?). It just has to
make sure things are synchronized so tools won't place a probe at
a point that has been removed already. And we should make sure that
all tools (gdb, systemtap, perf) can query it and agree on the format.
The only issue I see is that the current uprobes support in the linux
kernel is inode/offset based, which might make it a little harder to
probe these none-file-backed JIT code pages (except for gdb of course
which doesn't use uprobes directly).

One idea would be to add support for such a new service to libusdt
https://github.com/chrisa/libusdt and reimplement the hotspot JSDT
provider in terms of libusdt to make everything magically work on
all systems. And as added bonus make these more dynamic SDT probes
also magically work everywhere by just a simple recompile like
normal (DTRACE) SDT probes.

Cheers,

Mark