Re: Allowing multiple threads packages, a new threads package

[minyard at acm dot org]

Roland McGrath <roland@frob.com> writes:

> An important distinction may not be clearly understood from what Ulrich said.
> 
> The details like lock sizes are not part of the general libc ABI.  That is
> why the <bits/libc-lock.h> declarations can be opaque in the !_LIBC case.
> The only visible parts of the ABI that have locks, like libio's FILE, use
> indirection so that the type and size of the lock data structure are not
> exposed in the ABI.  
> 
> Those details are only part of the "internal" ABI between libc and libpthread.
> 
> As Ulrich pointed out, libc internally is intentionally quite portable to
> different locking implementations at the source level.  You could very well
> write a different bits/libc-lock.h variant and build a libc with that.  If
> done properly, the resulting libc.so.6 would be completely compatible with
> the existing ABI used by programs that use only libc--it's just
> incompatible with the existing glibc's libpthread.so.0 binary and any
> programs that require it.

Yes, of course, the design of libc in this area is very clean.  The
trouble with this approach is that each threads library would require
it's own version of libc.so.6.  I don't think it's feasible to
completely switch to a new threads system for an entire computer, that
would kind of defeat the purpose.  If you accidentally used the wrong
libc (which would be quite easy) the program would run, but glibc would
be running without locks.  I don't know of a way to detect this
problem, since glibc has to be able to run without the threads
library.  For my project, it was also unacceptable because we needed
both threads packages but didn't have room on the target's filesystem
for two libcs.

> It would be straightforward to write a variant bits/libc-lock.h that uses a
> level of indirection and relies on explicit initialization via the shared
> library initializer.  It would be simple to make a variant libpthread that
> works with this and does the necessary allocation and initialization.  Then
> you would be able to build your own variant glibc that is wholly compatible
> with the existing libc.so.6 and libpthread.so.0 ABIs, but can also be used
> with a different pthread implementation instead.  Ulrich objects to making
> the default glibc use indirection and explicit initialization (and probably
> rightly so).  So people would have to install your variant glibc package
> instead of the normal one, and pay that overhead when they use libpthread,
> to also be able to use your alternative thread library for glibc's internal
> locks.

Ulrich correctly objects to using explicit initialization and .ctors
to initialize the mutexes.  I knew it was a quick hack, but I was torn
between that and the alternative (stated in the next paragraph).  But
there is a way to do it without paying the runtime price.  I would
expect the runtime price is pretty small in general since most of
these are only initialized at times like startup or file opening
(except for sprintf, which is done on every call, but it seems odd
that sprintf would use mutexes at all).

>From what I understood of Ulrich's last objection, he correctly says
that if you don't want to use explicit initialization, glibc is making
an ABI contract with all the threads packages that might use it.  You
would have to build two initializers for the opaque locks, one for the
non-recursive case and one for the recursive case, and one for each
other data structure, and they would have to correctly map to
linuxthreads, and all other threads packages would have to live with
it.  I personally don't consider that a very bad situation.  The
layout of the linuxthread mutexes is, well, "historical", but it's not
that bad, most everything is initialized to zeros.  If you needed to
break that contract, that means you are changing linuxthreads to break
its ABI contract, and it would be quite drastic to do that and would
require a major release.  Then all the other threads packages would be
broken, but again, it's a major release and that's probably ok.

> Now that I think about it, I think this is so simple to do that I will
> experiment with it a little myself.

Yes, most of the work was analysis and testing, the actual changes
didn't take so long.  And if I hadn't done the silly __libcThr thing,
and I had done initializers for the opaque data structures that mapped
to linuxthreads, the changes would have been a lot easier, because I
spent a lot of time figuring out why _dl_load_lock didn't work right.
I guess I don't have to say this, but feel free to look at my patch
and steal stuff.  If you come up with something, I can modify my
threads package to run on your changes so you can do some validation
on a different threads package.

-Corey