Re: Fw: [ltc-interlock] [RFC] draft NFS trace hooks (fwd)

[Tony Reix <tony dot reix at bull dot net>]

Hi Li,

Le lundi 14 août 2006 à 11:11 +0800, Li Guanglei a écrit : 
> Hi, Tony
> 
...

>   SystemTap provides an infrastructure to do a dynamic probing, which
> means no patches to Kernel, no recompile and no reboot is required.

I think such features will be very useful in the field, when customers
encounter critical problems and that stopping the machine or updating
the kernel to a new version is not an option.


>   Currently we don't have a separate design document but you could
> refer to the following links to get an overall picture of what we are
> doing:
>  http://sourceware.org/systemtap/overview.html
>  http://sourceware.org/systemtap/man5/lket.5.html
>  http://sourceware.org/systemtap/man5/stapprobes.5.html

These links do not provide me high level information;
However, this one:
http://sourceware.org/systemtap/documentation.html
contains very useful document to refresh/increase my knowledge of
SystemTap (I studied the features of DTrace some years ago ...). 


>  And I attached a list of what we want to trace for NFS. But we are
> not working on NFS itself and don't have much knowledge about it.

We are working since some years on NFSv4, mainly testing or adding IPv6.
So we have some understanding of the protocol, but sure we do not master
the protocol nor the code.


> This is why we sending out the NFS trace hooks for a review to make sure
> that we didn't miss some important trace points.

Up to know, I've seen only 2 answers from the nfs mailing list, from
Chuck Lever. But I do not see answers to the 2nd email of Chuck. Maybe
I've missed it.


>  I put some answers to your questions below.

> Please tell me if we missed some important functions that should be
> probed for NFSv4. But what Xuepeng sent out is only our first step of
> NFS trace hooks. The full list of our plan is in the attachment.

As I said before, we do not master NFSv4 internals, which are VERY
complicated. I think only Trond Myklebust, Bruce Fields and Chuck Lever
could really help you to check that these hooks are appropriate. But I
know that Trond, Bruce and Chuck are very busy ... So they probably have
no time to spend to understand SystemTap goals and basics and to check
that the hooks you designed are appropriate.

Chuck asked for a design document. It seems you do not have one. And
building such a document would require a deep knowledge of NFS/NFSv4
internals. So this design document should be written either 1) by means
of a collaborative work between SystemTap and NFSv4 experts, or 2) by
someone who could spend time to build skills on both technologies, and
then discuss with the SystemTap and NFSv4 experts.


> >  > I know very few about SystemTap. However, I have questions:
> >  > - Do you have a design document describing what must be traced in NFS ?
> 
> See the attachment.

Hum. This looks more as a conclusion than as a design explaining the
rationale of the choices.


> >  > - How do you plan to submit your code to NFS or Kernel maintainers ?
> 
> No. The trace hooks written in tapsets is for dynamic tracing, which
> means that no patches to Kernel is needed.
>    
> >  > - How much of your code must be included in NFS code, as patches ? 
> 
> None. This is the powerful aspect of SystemTap: you can probe the
> Kernel without touching the Kernel source codes.
> 

Ohh. I did not know.
Probably this is explained in the OLS'06 paper.
Do you have a link to a technical explanation of how that works ?


> >  > - Have you worked with them (I saw nothing on mailing list) ?
> 
> No. we now work with SystemTap community. NFS is only a part of what
> we are doing.

I do not remember someone talking about systemtap in the nfs and nfsv4
mailing list. A quick search in the 3200 messages I've kept in my
folders seems to show that no one talked about systemtap before you did.

Do you know what is the position of OSDL about SystemTap ?


> >  > What are your plans about tapsets for NFSv4 ?
> 
> only NFSv4 procedure stub functions for the client and server side.
> 
> >  > Do you have resources to do that in 2006 or in 2007 ?
> 
> oh. I am not sure. We are assigned to work for SystemTap and LKET :-)

I'm used to have students giving help on some projects (this year: NPTL
Trace Tool, and NFSv4 Administration/Security).
Do you think a student (best French Computer Science University, 5-6
months of work) could help ?


> >  > First step could be to check that NFSv4 developers are already SystemTap
> >  > enthusiastic. If not yet, one should discuss of that with them. It seems
> >  > very important to get their comments and approval for which features to
> >  > trace. The problem is that NFSv4 Server developer is overloaded ... so
> >  > it may take some time.
> >  > Also, there already are some trace code in NFSv4.
> 
> It seems to me that a lot of developers still not realized the
> existence or how SystemTap could facilitate their
> development/debugging of Kernel. We need more advertisement for
> SystemTap :-)

Yes. I've talked with 2 guys here working on ext3 and they have no clear
idea of how SystemTap could help them ... One of them (a ext3 and Xen
expert) did not know about SystemTap even.

I know that proposing debugging/tracing tools to expert/guru developers
is difficult: they are often more efficient without using such tools but
they often forget that such tools will greatly help maintenance in the
future and to understand problems in the field.

What do Linux gurus (Linus, ...) think about SystemTap ?
Now, it seems that SystemTap is only an initiative by RedHat, IBM, Intel
and Hitachi. People close to the customers.


> SystemTap wiki is a good place to share the experience:
> 
> http://sources.redhat.com/systemtap/wiki/HomePage

Yes. I've found useful documents.
Now, I need to find time to read them ...


> Feel free to let me know your suggestions and questions. Thanks.

What about a student working on tapsets for NFSv4 in 2007 ? After we
have got the position of NFSv4 developers about SystemTap, sure. I'll
ask them about SystemTap.

Regards,

Tony