glibc detected *** free(): invalid pointer

[Sripathi Kodi <sripathik at in dot ibm dot com>]

I am seeing the following message when I run a multi-threaded testcase. 

*** glibc detected *** free(): invalid pointer: 0xb7f9fb78 ***

I recompiled glibc to make it abort when it hits this message. From the core file, I see the following backtrace:

#0  0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7dd37d5 in raise () at ../string/bits/string2.h:1000
#2  0xb7dd5149 in abort () at ../string/bits/string2.h:1000
#3  0xb7e0740a in __libc_message (do_abort=2, fmt=0xb7ec8c64 "*** glibc detected
*** %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:145
#4  0xb7e0db3f in _int_free (av=0xb7ed3820, mem=0xb7f38b78) at malloc.c:5525
#5  0xb7e0deba in __libc_free (mem=0xb7f38b78) at malloc.c:3404
#6  0xb7f75a5d in ___tls_get_addr (ti=0xb7ed1e30) at ../sysdeps/generic/dl-tls.c:670
#7  0xb7ea9c7b in __libc_dl_error_tsd () at dl-tsd.c:53
#8  0xb7f73045 in _dl_catch_error (objname=0xb7f382d0, errstring=0xb7f382d4,
operate=0xb7ea9770 <do_dlsym>,
    args=0xb7f382d8) at dl-error.c:155
#9  0xb7ea98ce in *__GI___libc_dlsym (map=0xb7ef9510, name=0xb7f2044d
"_Unwind_Resume") at dl-libc.c:42
#10 0xb7f1f330 in _Unwind_ForcedUnwind (exc=0x0, stop=0, stop_argument=0x0)
    at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:44
#11 0xb7f1ce21 in __pthread_unwind (buf=Variable "buf" is not available.
) at unwind.c:130
#12 0xb7f18200 in __pthread_exit (value=0x0) at pthreadP.h:222
#13 0xb7f3d019 in internal_exit ()
   from /home/sripathi/ibm-java2-ws-sdk-50-linux-i386/jre/bin/realtime/libj9thr23.so
#14 0xb7f3c33f in thread_wrapper ()
   from /home/sripathi/ibm-java2-ws-sdk-50-linux-i386/jre/bin/realtime/libj9thr23.so
#15 0xb7f175c1 in start_thread (arg=0xb7f38ba0) at pthread_create.c:261
#16 0xb7e736fe in clone () from /lib/tls/libc.so.6

There seems to be some problem with glibc freeing some internal memory. This is what I have observed about the memory being freed:

mem being freed is 0xb7f9fb78, hence chunk is 0xb7f9fb70
(gdb) p *(struct malloc_chunk *)0xb7f9fb70
$3 = {prev_size = 0, size = 0, fd = 0xb7f393a0, bk = 0xb7f9fe00}

I have noticed that every single time this happens, there is just one more thread in the process, and that is doing _exit(). Is it possible that there is a race between a thread doing pthread_exit and another doing exit to tear down the whole process?

Any idea what could be going wrong? Any hints on how to debug this further?