Re: Notes from the systemtap BOF

[William Cohen <wcohen at redhat dot com>]

Maneesh Soni wrote:

> There was one more request (not discussed at BOF) related to IO layer
> tracing. 
>
> It is to get information about time taken in executing a probed routine. 
> There should be some way to find out how much time (could be CPU cycles or 
> human time) is taken in the executions of various routines in the code 
> path. It could be useful in analysing IO stalls / hangs etc. 
>
> I don't know if that is possible in systemTAP or not
> but looks a nice feature to me.
>
>
> Thanks
> Maneesh

There is interest in getting time measurements. However, this can be 
complicated. There is a range of time scales and resolutions that the 
instrumentation could be interested in, e.g. milliseconds and 
nanoseconds/clock cycles.

On an SMP machine there is nothing to keep the time stamp counters in 
step and there can be skew in the values. Thus, it would be possible for 
an event to happen on one processor A, its timestamp counter be 
recorded. The related event happens on processor B; its timestamp 
counter be recorded. Due to skew the raw data could make it appear that 
B happened after A. Or B happen after A. Or B happens WAY after A.

On some processors the cycle counter is literally that, counting 
processor cycles. Thus, when the clock speed is changed the rate the 
register is incremented changes, making it difficult to get the real 
time from the cycle count as the processor clock is changed.

It would be really nice if the hardware designers provided a machine 
specific register that provide high resolution clock. This would make 
life so much easier to merge data together and create reasonable 
timelines and determine cause and effect. A sub-$100 consumer GPS can 
very accurately determine the time. Why can't a regular processor? A 
second system setable one for virtual time would also be useful.

My understanding is that LTT has some support to take into account the 
possible skew between the process time stamp counters.

-Will