sleeping, locks and debug kernels

[Mark Wielaard <mjw at redhat dot com>]

Hi,

While adding the support for "dynamic" vma registration it really helped
to have a debug kernel around that had the various LOCK debugging
configs set. Especially CONFIG_LOCKDEP is pretty handy. Some distros
have a separate debug kernel around that you can (should) install when
hacking on systemtap to catch issues early (fedora has yum install
kernel-debug, after booting into it just run stap-prep to get all
necessary -devel and -debuginfo packages).

This will slow down your system a bit, but will show when code possibly
tries to sleep while in interrupt context (where sleeping is really
fatal), some accesses to resources that don't have the required locks,
threads taking locks in different order that could possible deadlock and
when code holds a lock while calling a sleeping function.

Sadly our current code is somewhat noisy because it has some of this
issues. I resolved the ones in uprobes where we weren't holding the
rcu_read_lock where necessary. And the one in the transport layer, where
we were holding a mutex over a lot of code that could potentially sleep.
Reviews of commit 262f75 and commit ab8633 appreciated since any changes
in the locking code is always a little scary.

There is one issue I don't know how to solve. That is
stap_start_task_finder() this takes a rcu_read_lock() goes over every
task, inspects each, calls utrace_attach on it if appropriate, gets the
task->mm, adds the engines to some internal datastructures, checks that
unprivileged users don't get access to utrace engines of task that
aren't theirs and then after doing that for each task releases the lock.
The problem is that utrace_attach_task() may sleep, since it must
allocate memory to create a new enginer. Which is not nice while we have
the rcu_read_lock. But I don't immediately see how to split up this loop
so that we only hold the lock while doing non-sleepy things.

Maybe task_finder2 will resolve all these issues, but we probably will
be supporting utrace and task_finder[1] for some time. Besides being a
little bad to sleep while holding that lock, and making dmesg very
noisy, it also prevents lockdep from being more useful. lockdep will
show the which locks were taken when it encounters something bad, but
only for the first such issue. Which now is almost always the
stap_start_task_finder() call. So if we could solve this one, then
running under a debug kernel would be much more useful.

Anybody got a good idea how to refactor stap_start_task_finder() to make
lockdep happy?

Thanks,

Mark