Evaluating SystemTap for Network Response Times

[Nathan DeBardeleben <ndebard at lanl dot gov>]

I'm looking at using SystemTap and/or [dj/k]probes to time network 
operations inside the kernel.  Specifically, we want to time the point 
when a socket send operation leaves user space, entering kernel space, 
down to the point where the kernel says "it's done, sent".  Obviously 
this may involve fragmentation of packets so we'd need some way to keep 
track that these N fragments came from this initial operation, and as 
they close up, we know we're not completely done until all fragments are 
done.

Initially this looks just like the kind of thing I could do with 
SystemTap but I worry that the scripting language will be too 
restrictive to allow me to allocate these types of data structures to do 
record keeping.  When it comes down to it - I want to observe a system 
and recognize outliers ("hey, this operation took 20 times longer than 
the rest") through statistical means.

I was hoping I could get some feedback from the SystemTap 
users/developers as to whether (1) this seems feasible, (2) SystemTap 
seems like the appropriate tool, and (3) perhaps if anyone is aware of 
similar projects.

I will be experimenting with this in a parallel computing environment 
and with single system image tools such as bproc and the brand new 
XCPU.  I hope I can add some value to the SystemTap community by testing 
it out in these environments.  If this first step goes well, I will be 
looking at using SystemTap for monitoring parallel file systems and 
studying potential performance bottlenecks.

Thanks for your time.

--
-- Nathan
Correspondence
---------------------------------------------------------------------
Nathan DeBardeleben, Ph.D.
Los Alamos National Laboratory
Parallel Tools Team
High Performance Computing Environments
phone: 505-667-3428
email: ndebard@lanl.gov
---------------------------------------------------------------------