Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Paul Pluzhnikov <ppluzhnikov at google dot com>]

On Sat, Nov 1, 2008 at 8:28 AM, Andrew Lofthouse wrote:

> I'm going to be getting a bunch of spam, now, huh?

This wasn't the first message so quoted.

I find gmail spam filtering to be extremely effective:
I only get about 1 spam/month through.

>> One thing that may have changed is prelink addresses --
>> IIRC, Fedora runs prelink at 2 week intervals.
>
> Okay, it must have been prelink.

Indeed. I did '/usr/sbin/prelink -ua', and now I can reproduce
this on the thread-so test case (with 32-bit kernel).

Reproduces with current GDB CVS head...

The problem is a (now almost 3 year old) bug in glibc:
http://sources.redhat.com/bugzilla/show_bug.cgi?id=2328
for which a patch has been available for 2 years, but came
from the "wrong" person :-(

In this case, GDB gets notified about libpthread load
before libpthread has been relocated:

GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(gdb) r
Starting program: /tmp/a.out
[Thread debugging using libthread_db enabled]
Error while reading shared library symbols:
Cannot find new threads: generic error
Cannot find new threads: generic error
(gdb) bt
#0  0x0011f1a1 in _dl_debug_state () at dl-debug.c:76
#1  0x00122a2b in dl_open_worker (a=0xbfc4a9ec) at dl-open.c:366
#2  0x0011e4f6 in _dl_catch_error (objname=0xbfc4aa14,
errstring=0xbfc4aa10, mallocedp=0xbfc4aa1b,
    operate=0x122820 <dl_open_worker>, args=0xbfc4a9ec) at dl-error.c:178
#3  0x001222f2 in _dl_open (file=0x8048574 "./thread-so.so",
mode=<value optimized out>, caller_dlopen=0x8048479, nsid=-2, argc=1,
    argv=0xbfc4ac64, env=0xbfc4ac6c) at dl-open.c:596
#4  0x0012fc2c in dlopen_doit (a=0xbfc4ab80) at dlopen.c:66
#5  0x0011e4f6 in _dl_catch_error (objname=0x133058,
errstring=0x13305c, mallocedp=0x133054, operate=0x12fb90
<dlopen_doit>,
    args=0xbfc4ab80) at dl-error.c:178
#6  0x0013005c in _dlerror_run (operate=0x12fb90 <dlopen_doit>,
args=<value optimized out>) at dlerror.c:164
#7  0x0012fb61 in __dlopen (file=0x8048574 "./thread-so.so", mode=1)
at dlopen.c:87
#8  0x08048479 in main () at thread-so-main.c:6
(gdb) p &stack_used
$1 = (list_t *) 0x2ba110
(gdb) x/x $1
0x2ba110 <stack_used>:	0x00016110
(gdb) shell nm /lib/libpthread.so.0 | grep stack_used
00016110 d stack_used   ### NOTE: stack_used points to
                        ### non-relocated (and unreadable) self!

(gdb) fr 8
#8  0x08048479 in main () at thread-so-main.c:6
6	 void *h = dlopen("./thread-so.so", RTLD_LAZY);
(gdb) b 7
Breakpoint 1 at 0x804847c: file thread-so-main.c, line 7.
(gdb) c
Continuing.
[New Thread 0xb803e6c0 (LWP 32044)]
[Switching to Thread 0xb803e6c0 (LWP 32044)]

Breakpoint 1, main () at thread-so-main.c:7
7	 void (*fn)(void) = dlsym(h, "foo");
(gdb) x/x $1
0x2ba110 <stack_used>:	0x002ba110   ### NOTE: stack_used now points to
relocated self.

The 'stack_used' a head of thread list, and libthread_db can not
work correctly if it observes the wrong (unreadable) value.

I don't know what it takes to make Uli apply the patch to fix this :(
Perhaps Tom can have a friendly chat with him?


Cheers,
-- 
Paul Pluzhnikov