Re: Performance problem of dlopen

[Joerg Budischewski <Joerg dot Budischewski at sun dot com>]

Hi again,

> Had to modify the list (are you using GCC 3.0 and not 3.2 BTW?),
Yes. OpenOffice is lazy with switching the compiler, because we (the offical
OpenOffice and StarOffice build) want to uphold binary compatibility,
so that binary 3rd party extensions still run.

> Anyway, the difference is clear. If you load libwrp641li.so directly,
> the
> symbol lookup path is set immediately to all the libs loaded because
> of it,
> with the required breadth search, so say libtl641li.so's
> _ZTS11INetIStream
> symbol is looked in:
> 31446:  symbol=_ZTS11INetIStream;  lookup in file=/tmp/x
> 31446:  symbol=_ZTS11INetIStream;  lookup in file=/lib/libdl.so.2
> 31446:  symbol=_ZTS11INetIStream;  lookup in
> file=/usr/lib/libstdc++.so.5
> [...]
>
> while if you dlopen the libraries one by one (preferrably in the
> actual dependency order, e.g. as shown
> by LD_LIBRARY_PATH=. prelink -nv ./libwrp641li.so), then on average
> you have
> half as big symbol lookup path (starts from 3 or how many libs and
> grows
> till all libs), the above symbol goes through:
>
> 31498:  symbol=_ZTS11INetIStream;  lookup in file=/tmp/x
> 31498:  symbol=_ZTS11INetIStream;  lookup in file=/lib/libdl.so.2
> 31498:  symbol=_ZTS11INetIStream;  lookup in
> file=/usr/lib/libstdc++.so.5
> [...]
> There is one thing which ld.so can do better (it is even implemented
> in ld.so, but disabled for now because e.g. librt.so depends
> on the old behaviour)
When would you guess can and will this be switched ?

> - ATM if symbol lookup encounters a weak symbol,
> it keeps searching if it finds a strong symbol. If it does not, it
> uses
> the first weak symbol found, otherwise it uses the strong symbol.
> The currently disabled behaviour is stop when it finds first weak or
> strong symbol.
The different behaviour for implicit and explict loading of libraries 
remains
strange to me, but I believe you that this is elf spec.

For comparison, I e.g. tested with the solaris loader which showed
identical behaviour for strong symbols, but uses the optmized
lookup method for weak symbols as you described.

So as far as I see, there are two disadvantages coming together, the 
currently
used weak lookup strategy and the fact, that gcc3 makes so much use of 
weak symbols.

So if I want to optimize this without your static-link suggestion  (I 
will come
to this later on), it seems to be a good approach to find a way to get rid
of these weak symbols. We need them for C++ gcc3 exception handling (rtti
symbols for the same exception must be identical), however I think vtables,
etc. could be filtered.

> But even when this is changed, the symbol lookup when loading the libs
>  one
> by one in the right order will be always faster, since on average the
> symbol
> lookup path will be shorter.
Maybe a little nitpicking, but couldn't ld.so do this (e.g. it knows also
the correct order for calling the init functions) ? I must admit, that
I didn't understand the search order, that is chosen by dlopen 
currently. Is it
well defined or has ld.so there some freedom (if it is well defined, can
you maybe give me a pointer to a documentation, maybe this can be tweaked a
bit with e.g. a better link order ?)

> What would IMHO help most for OpenOffice startup spent in the dynamic
> linker
> (in addition to aggressive profiling and speeding up often used
> functions)
> is to merge shared libraries which are
> a) linked statically to the OOo programs
> b) dlopened during start of every OOo program (or at least all the
> important
>    ones; IMHO spadmin or setup startup performance is not critical,
> while
>    soffice with the usual invocations is)
> into libOpenOffice.so and link it into soffice etc. programs
> statically.
Hmm, the problem for us is how to make a reliable build process out of it.
In general, every used static library leads to consistency errors in our
build env as we tend to do frequent respins only on modified projects,
especially when we get nearer to a golden master.  So introducing lots
of static libs definitly is a problem for us.

If I understood you correctly, we would have the same compiled code
within shared libraries (to e.g. serve other executables like setup,
etc. ) and the office executable. This would also lead to a higher memory
footprint because duplicated code gets into the process (e.g.
by some optional libraries, which are still linked to the shared libraries).

Another problem is, that there are semantics in the library names (e.g.
every UNO components export one identical named function, which allows
to instantiate objects within a library).

And semantically, we head for more modularization, going the reverse
direction is, hmm,  not so nice.

But nevertheless it is an interesting option, we will certainly discuss
this.

>> > That way you kill lots of duplication (which means less
>> > relocations,
>> > less
>> > memory used and fewer dynamic symbols) and also prelink(8) can
>> > speed
>> > it
>>Where can I get prelink ?
>
>
> ftp://people.redhat.com/jakub/prelink/
> You need a recent glibc (like 2.3 or never, 2.3.93 in RHL 8 should be
> enough too) for it too.
We of course still need to run also on 2.2.x glibcs.

However, this might e.g. be an alternative for linux distributors and
OpenOffice (or for having a separate executable mmm...). I will give it 
a try,
I need to find a machine  with a redhat 8.0 installation.

>
> If all the /usr/lib/openoffice/program libs in the above list were
> merged into one, I think the number of conflicts (3870) could go
> down substantially (and thus even speed things up).
yep, I am aware of this.

Thanks for your information, this was a big help. At least we currently 
know,
why we have these performance problems compared to solaris.

Thx for your time,
Joerg