Re: Per-process tracing user-space probes approach

[Richard J Moore <richardj_moore at uk dot ibm dot com>]

Prasanna, what's the long-term plan to avoid COW scalability issues?
- -
Richard J Moore
IBM Advanced Linux Response Team - Linux Technology Centre
MOBEX: 264807; Mobile (+44) (0)7739-875237
Office: (+44) (0)1962-817072


                                                                           
             "S. P. Prasanna"                                              
             <prasanna@in.ibm.                                             
             com>                                                       To 
             Sent by:                  systemtap@sources.redhat.com        
             systemtap-owner@s                                          cc 
             ourceware.org                                                 
                                                                   Subject 
                                       Per-process tracing user-space      
             09/06/06 06:53 AM         probes approach                     
                                                                           
                                                                           
             Please respond to                                             
                 prasanna                                                  
                                                                           
                                                                           




Hi,

I have listed a brief description of the user-space probes approach,
which I am planning to implement.
Please review and provide your comments.

Thanks
Prasanna

Requirements:

- per process tracing using "COW"
- able to trace yet to be started applications
- smallest kernel patch (no aggregate support for this release)
- correlation of kernel and user probes output
- least performance overhead compared to ptrace
- provide clean user interface like syscall with pre-defined
  set of handlers that can log data, registers, stack trace etc and
  also support adding new handlers at runtime.
- no hooks to readpage(s)
- handler runs in kernel and handler can sleep to collect data from
  non-memory resident pages.
- single step out-of-line

Usage:

1. Specifying a new kernel handlers for a probe point.
             - usage example.
                1. Trace a library routine malloc() in an application
already
                   started pid is 123. A kernel module with
uprobe_khandler() is
                   inserted into the kernel.

                 To get the address of malloc() use
                 #objdump -D appln |grep malloc
                 0x08048320
                 main(){
                         pid_t child = 123;

                         printf("insert probes on child pid %ld\n", child);
                     utrace(child, 0x8048320, UTRACE_KHANDLER,
uprobe_handler);
                         }

                 2. Trace a routine foo() in an application yet to be
started and
                    specify a kernel handler utrace_foo_khandler().
                    First of all a kernel module with utrace_foo_khandler()
is
                    inserted into the kernel.

                    To get the address of foo() use
                    #objdump -D appln |grep foo
                    0x08048fa0

                    main(){
                                     pid_t child;
                                     if ((child = fork()) == 0) {
                                                 utrace(0, 0x08048fa0,
UTRACE_KHANDLER, utrace_foo_khandler);
                                                 execve(appln,
"/home/prasanna/appln");
                                     }
                         }

2. Specifying already existing kernel handler for a probe point.
             - usage example.
                1. Trace a library routine malloc() in an application
already
                   started pid is 123 and specify to dump registers.

                 To get the address of malloc() use
                 #objdump -D appln |grep malloc
                 0x08048320
                 main(){
                         pid_t child = 123;

                         printf("insert probes on child pid %ld\n", child);
                     utrace(child, 0x8048320, UTRACE_GETREGS, NULL);

                 }

                 2. Trace a routine foo() in an application yet to be
started and
                    specify to dump registers.

                    To get the address of foo() use
                    #objdump -D appln |grep foo
                    0x08048fa0

                    main() {
                                     pid_t child;

                                     if ((child = fork()) == 0) {
                                                 utrace(0, 0x08048fa0,
UTRACE_GETREGS, NULL);
                                                 execve(appln,
"/home/prasanna/appln");
                                     }
                         }

Issues:

1. Is it acceptable to allow the user to specify a kernel
   routine through a syscall that will be executed, when the probe
   point gets hit ?
2. Is it acceptable to run the instrumentation code as part of kernel
   address space ?
3. Are there any security concerns ?

Interfaces:

int sys_utrace(pid_t pid, unsigned long vaddr,
                                     unsigned long request, char *name);

 pid                     - process id that need to probed.
 vaddr                         - virtual address where probe is to be
inserted.
 request           - UTRACE_GETDATA, UTRACE_GETREGS, UTRACE_STACKTRACE.
 name                    - name of the kernel handler.

maybe _add_ length field as well, so that user can specify length of data
to be logged.

void sys_utrace_rm(pid_t pid, unsigned long vaddr);

Data structures:

Allocated for each probe.
struct uprobe {
             /*per process and per probe hlist_node */
             struct hlist_node plist;
             unsigned long request;                    /* bitmap of the
request */
             unsigned long status;                     /* status as
active/inactive */
             struct kprobe kp;                         /* kprobe structure
*/
};

Allocated for each process.
struct uprobe_module {
             struct hlist_head phead;            /* list of all probed
processes */
             /* list of all probes for individual process */
             struct hlist_node mlist;
             struct pid_t pid;                         /* pid of the each
probed process */
};

uprobe_table[];                      /* individual probes hashed on vaddr *
pid */
struct hlist_head uprobe_module_head[];
                                     /* list of all uprobe_module hashed on
pid*/
uprobe_mutex                         /* protect uprobe_table and
uprobe_module_table*/
--
S.P. Prasanna
Linux Technology Center
India Software Labs, IBM Bangalore
Email: prasanna@in.ibm.com
Ph: 91-80-41776329