[RFC] multithreaded bfd?

[DJ Delorie <dj at redhat dot com>]

Been thinking about how to speed up the linker some more.  Today's
thought is to use multiple threads to take advantage of dual core
processors that seem to be becoming popular.

I figured there's a couple of steps that could be parallelized,
assuming we can figure out a way around the I/O issues, such as using
memory mapped input and output files so that we can safely do I/O in
parallel.

* Reading in the input BFDs.

* Resolving undefined symbols.

* Applying relocations.

* Writing out the BFDs to the output BFD.

I've come up with a simple API that can be use for with-threads and
without-threads, in simple situations.  I don't want to recreate a
full POSIX threading api (might as well just use pthreads), just
something trivial that buys us a lot of processing power.  The theory
is, all you really need is a way to start N threads from a loop, then
wait for them to finish.  Add a simple exclusive lock mechanism, wrap
some common functions in a lock/do/unlock (like malloc/free), and
we're there.

I've written a simple thread API implementation that works without
threads (i.e.  linear) and with pthreads, documentation is attached.
Not sure exactly where to start plugging this in, though.

So, what does everyone else think of this kind of idea?

----------------------------------------------------------------------

@deftypefn Extension void djthread_start (thread_func @var{func}, void *@var{data})

This function runs the given function, possibly in its own thread,
possibly immediately.  The given @var{data} is passed to the function.
If threads are not supported, @var{func} is called directly, and will
complete before this function returns.  If threading is supported,
this function will not return until @var{func} is started.  Note that
there may be a limit to the number of threads that may run; if so,
this function waits until some other thread finishes before starting
@var{func}.

The intended use is to have the main thread loop over some actions
which may be (but need not be) executed in parallel, then call
@code{djthread_wait} right after the loop.

@example
  for (i=0; i<n_sections; i++)
    djthread_start (apply_relocs, section[i]);
  djthread_wait ();
@end example

Note that this function should only be called from the main thread.

@end deftypefn

@deftypefn Extension void djthread_wait ()

This function waits for all started threads (i.e. everything except
the main thread) to finish.

Note that this function should only be called from the main thread.

@end deftypefn

@deftypefn Extension void djthread_lock (void *@var{ptr})

This function is used to provide mutually exclusive locks for threads
to use to coordinate access to shared variables.  Ideally, you should
pass a pointer to the variable you're about to test/modify, then
test/modify the variable, then pass @code{NULL} to this function to
release the lock.

Note that you cannot lock more than one object at a time.  To lock a
group of objects, designate one of them to be used for this purpose,
or create a new variable just for this purpose if no suitable variable
is available.

Note that some implementations may not support distinct locks; you
must assume that two threads locking @emph{different} objects may act
as if they locked the same object (i.e. the implementation may only
care about NULL vs non-NULL pointers, and ignore the actual value).
However, some may, so design for fine grained locking but allow for
global lock implementations too.

So, keep the lock for the minimum amount of time, and unlock it as
soon as possible.

Note that this function may not be called from the main thread, it may
only be called from threads started by @code{djthread_start}.

@end deftypefn