Re: Performance problem of dlopen

[Jakub Jelinek <jakub at redhat dot com>]

On Thu, Oct 24, 2002 at 11:33:29AM +0200, Joerg Budischewski wrote:
> My name is Joerg Budischewski, I am working on the OpenOffice project
> (see http://www.openoffice.org).
> 
> I have some questions to the performance behaviour of the dlopen
> function, I am currently using libc 2.2.4 on my linux x86 system.
> 
> During investigations of the office startup I had a closer look at
> loading shared libraries. Investigating this further, I realized a (at
> least to me) surprising behaviour.
> 
> Using one dlopen() to load a shared library A (which is linked to
> different other shared libraries [B-X]) is significantly slower than
> using multiple dlopen() calls  for loading [B-X] before A.
> 
> In numbers, on my system the library takes 3300ms to load with one
> dlopen, while using multiple dlopen takes 2100ms. I have actually played
> a little around with LD_DEBUG=symbols and found out, that in the first
> case, lookup for symbols which are located in dependent libraries are
> searched also in the higher libraries, which leads to significantly more
> lookups.
> 
> So my question is if this is a desired behaviour or can this be
> considered as a performance bug ?

I'd prefer to see your testproglets first. ld.so has to obey ELF
lookup rules, so it is questionable whether anything can be done here.

What would IMHO help most for OpenOffice startup spent in the dynamic linker
(in addition to aggressive profiling and speeding up often used functions)
is to merge shared libraries which are
a) linked statically to the OOo programs
b) dlopened during start of every OOo program (or at least all the important
   ones; IMHO spadmin or setup startup performance is not critical, while
   soffice with the usual invocations is)
into libOpenOffice.so and link it into soffice etc. programs statically.
That way you kill lots of duplication (which means less relocations, less
memory used and fewer dynamic symbols) and also prelink(8) can speed it
up a lot (prelink only speeds things up for libs linked to a program
directly, not dlopened ones). When I was looking at it last time, there
could be up to 55 libraries merged into libOpenOffice.so if we allowed
spadmin and setup to link against some code it will not use, if not, I think
still something like 30 or so libraries would apply anyway.
It would require quite minimal changes in OOo module loader (basically,
have the list of modules stored in libOpenOffice.org compiled in and do
nothing if such module is requested, plus for modules which have a control
structure looked up with dlsym hack up code to allow more such structs
in one library), plus of course some infrastructure changes to build the big
library (perhaps in addition to) the little libraries.
Due to time constraints I got only to a point of gathering such libs
and writing a script to build such library after OOo build finishes,
had not had time to actually hack up the OOo module loader changes.

> Another point which I am interested in is that since we are using gcc
> 3.x for building, loading shared libraries lazy is significantly slower,
> one main reasons seems to be that unused symbols get relocated with gcc3
> which have not got relocated with gcc2. When I use nm on both libraries,
> the output (beside name mangling of course) looks similar. Does anyone
> have an idea, what this is caused by ?

The C++ ABI changed and unfortunately it is not exactly startup time
friendly :(.

> Thx for your comments in advance,
> 
> Joerg
> 
> PS: If someone is interested in reproducing the behaviour with an
> OpenOffice installation set, just let me know, I can provide some
> testprograms for it.

Please.

	Jakub