GDB support for thread-local storage

[Jim Blandy <jimb at redhat dot com>]

I'd like to extend GDB to support thread-local variables.  Richard
Henderson, Ulrich Drepper and I have come up with some pieces of the
solutions; pretty much everything outside GDB has been settled, but
I'm still trying to figure out how GDB will pull the pieces together.

This post describes:
- the feature we're trying to support,
- the parts we've worked out so far, and
- the parts in GDB that I'm still trying to sort out.

Some implementations of C and C++ support a ``__thread'' storage
class, for variables that occupy distinct memory in distinct threads.
For example, the definition:

       __thread int foo;

declares an integer variable named ``foo'' which has a separate value
and address in each thread, much as a variable declared ``auto'' has a
separate value and address in each invocation of the function
containing its declaration.  Creating a new thread creates a new
instance of ``foo'', and when the thread exits, the storage for
``foo'' is freed.

Typically, a program includes an ``initialization image'' --- a block
of memory containing the initial values for any thread-local variables
it defines.  When the program creates a new thread, the run-time
system allocates a fresh block of memory for those thread-local
variables, and copies the initialization image into it to give the
variables their initialized values.

A dynamically loaded library may also define thread-local variables.
Some implementations delay allocating memory for such variables until
the thread actually refers to them for the first time.  This avoids
the overhead of allocating and initializing the library's thread-local
storage for all the threads present in a program when the library is
loaded, even though only a few threads might actually use the library.

Thread-local storage requires support in the ABI, and support in the
dynamic linker, if you want reasonable performance.  There's a
complete description of how it's done on the IA-32, IA-64, and SPARC
at http://people.redhat.com/drepper/tls.pdf.  This is based on
specifications already written for the IA-64 and SPARC; I think the
IA-32 implementation is Ulrich Drepper's work.

For GDB, the first question is: how should the debugging information
describe the location of a thread-local variable?  We generally answer
this sort of question by looking at how the code generated by the
compiler finds the variable, and then emitting debugging information
that matches that.

To allow the run-time system to allocate thread-local storage on
demand, the ABI in certain circumstances requires the compiler to emit
a call to a function, __tls_get_addr, to find the address of a
thread-local variable for the current thread and a particular module.
This function looks up the address in a table, allocates and
initializes the storage if necessary, and returns its address.

Unfortunately, Dwarf 2 location expressions cannot perform function
calls in the inferior.  We could extend it to do this, but inferior
function calls are rather complicated (look at the *_push_arguments
functions and hand_function_call in GDB); I don't think this is a good
idea.

Instead, I've suggested adding a new Dwarf 2 opcode: 

    12. DW_OP_push_tls_address

    The DW_OP_push_tls_address operation pushes the base address of the
    current thread's thread-local storage block.  If the expression occurs
    in the Dwarf information for a dynamically loaded library, then
    DW_OP_push_tls_address pushes the base address of that library's block
    for the current thread.  If the library's storage for the current
    thread has not yet been allocated, a Dwarf consumer may arrange for it
    to be allocated now, or report an error to the user.

When an implementation allocates thread-local storage on demand, this
makes it hard to describe the location of a thread-local variable
using ordinary Dwarf expressions: referencing the storage may entail
allocating memory, copying an initialization image into place,
registering it with the thread, and so on.  A dedicated operation like
DW_OP_push_tls_address leaves this complicated task to the debugger,
which is presumably already familiar with the program's ABI and thread
system, and can handle the request appropriately.

I've posted a note to the Dwarf mailing list, describing the
DW_OP_push_tls_address approach, and saying that we'll experiment with
this as a GNU extension to Dwarf and write back when we've actually
got something working.

For STABS, we can simply invent a new symbol type, whose value is the
offset within the thread-local storage block for the current thread
for the module containing the stab.  I haven't written up a real
proposal for STABS yet.

On Linux, Ulrich Drepper has added the following function to
libthread_db:

/* Get address of thread local variable.  */
extern td_err_e td_thr_tls_get_addr (const td_thrhandle_t *__th,
                                     struct link_map *__map, size_t __offset,
                                     void **__address);

This takes a thread handle, an entry from the dynamic linker's link
map, and an offset, and sets *__address to point to the base of that
thread and module's thread-local storage, plus the offset.  It returns
an error code if the space hasn't been allocated yet.

Note that this interface is not cross-debugging clean.  Actually, none
of the libthread_db interface is --- the underlying proc_service
interface uses the hosts' `paddr_t' type to represent addresses in the
running program, and the top-side interface uses `void *' to represent
the location of thread-local data, and in the function above.  More on
this later.


So now we get to the part which isn't really sorted out yet, in my
opinion.  There's a lot that needs to happen between the point where
GDB reads debugging information for a thread-local variable, and the
point where GDB can find a thread-local variable's value.

GDB already has an address class apparently intended for describing
thread-local data.  In symtab.h:

    enum address_class
      {
        ...

        /* Value is at a thread-specific location calculated by a
           target-specific method. */

        LOC_THREAD_LOCAL_STATIC,

        ...
      };

It would be more proper for that comment to say that the location is
calculated by an ABI-specific method.  There needs to be a new gdbarch
method:

  /* return a `struct value' for the object of type TYPE at OFFSET in
     the thread-local storage block for THREAD and MODULE.  If the
     thread-local storage block hasn't been allocated yet, raise an
     error.  */
  struct value *gdbarch_tls_get_value (struct thread_info *thread,
                                       struct objfile *module,
                                       LONGEST offset,
                                       struct type *type);

In order for this to reach the thread layer where we can call
libthread_db, there will need to be a new target stack method as well,
which the gdbarch method can invoke if it pleases.  Something similar
to the gdbarch method:

    struct value *(*to_tls_get_addr) (struct thread_info *thread,
                                      struct objfile *module,
                                      LONGEST offset,
                                      struct type *type);

The default implementation will raise an error, saying we don't know
how to find thread-local storage on this system.  The libthread_db
target will override that default with something that calls
td_thr_tls_get_addr.

If you're not convinced it should be a target method, consider this:
Remember that libthread_db isn't clean for cross-debugging.  It's a
target library.  So at the moment, there are cases where gdbserver
loads and uses libthread_db, not GDB itself.  In those cases, the
tls_get_addr request needs to be sent across the network connection to
gdbserver, td_thr_tls_get_addr needs to be invoked there, and the
answer needs to be sent back.  By making tls_get_addr a target method,
it's easy for the remote protocol layer to provide its own definition
of the method and send a packet across for the request.