Java profiling with systemtap

[Peter Süttner <Peter dot Suettner at mailbox dot tu-dresden dot de>]

Hi,

I am currently involved in a project called CoolSoftware (you can refer 
to www.*cool-software*.*org*/ for more information). Our goal is to 
optimize  energy consumption of overall IT infrastructures by developing 
a reconfigurable, energy-adaptive application architecture.

In order to achieve this goal, it becomes necessary to analyse the 
utilization of physical resources caused by specific applications. At 
this point, systemtap comes into play. It seams to be very suitable for 
profiling the resource usage of a specific application (e.g. CPU 
utilization, disk IO, network traffic, ... ), which works fine.

In particular, I am currently working at a resource usage profiler for 
Java applications. We want to profile at the interface-level of a 
component, that is a simple method at the moment. I am aware of the 
hotspot probes added to the JVM, but there are several problems with these:

1. To trace methods in Java, -XX+ExtendedDTraceProbes flag needs to be 
set at JVM start up.  They have a very high overhead, since probes seem 
to fire at every method call (and there are really a lot of them) made 
within the JVM process. I couldn't find a way to filter for specific 
method. Any suggestions?
(we want to be able to profile production systems, so the overhead is a 
real problem )
2. If the hotspot tapset is used, we are bound to a specific JVM, since 
the probe points are in the shared library of the JVM, aren't we?

So far, I am using a workaround. I am using bytecode instrumentation to 
inject something like
'System.out.println("Package.Class.Methodname.method_entry")' at method 
entry,
and
'System.out.println("Package.Class.Methodname.method_exit")' at method 
return (only for methods we are interested in)
This causes a write() syscall and it's argument is parsed in a systemtap 
script. In combination with the TID, it should be possible to connect 
specific resource usage to the currently executed method. That way I can 
keep track of a specific subset of methods and it is still independent 
of a specific JVM.

Surprisingly, this works quite well for small examples (haven't tested 
big ones yet). But it seems to be a very unclean solution. So I decided 
to ask you guys. Maybe you know about a better solution....

Any suggestions, comments, feedback would be appreciated.

best regards,

Peter