Using readahead in ld.so for 10% startup gain in big apps

[Lorenzo Colitti <lorenzo at colitti dot com>]

Hi,

after my hackish readahead experiments, I modified ld.so to call 
readahead(2) on libraries immediately after mmaping them into memory. 
This yields a > 10% gain in startup times for big apps. For example, 
GNOME startup goes from 28 seconds to 25 seconds [1] and openoffice from 
25 seconds to 22 seconds on cold boot.

The attached one-line hack calls readahead(2) unconditionally. I realize 
that (a) this is Linux-specific and (b) it might not be a good idea to 
call readahead unconditionally on all libraries (although it might well 
be). However, I think this would make sense to have this in ld.so at 
least as an option which could be turned on at library compile time.

What would have to be done to include this feature in trunk ld.so? Would 
I have to implement a flag in the ELF header? If so, any pointers?


Cheers,
Lorenzo

P.S. Steve suggested [2] that I use madvise(), but when I tested that it 
seemed to have no effect at all. This is probably because the kernel 
interprets WILLNEED and/or SEQUENTIAL simply as advice and does not act 
on it.


[1] See bootcharts at:
http://www.gnome.org/~lcolitti/gnome-startup/analysis/ld-nopatch.png
http://www.gnome.org/~lcolitti/gnome-startup/analysis/ld-patch.png

[2] http://sources.redhat.com/ml/libc-alpha/2005-09/msg00020.html
--- glibc-2.3.5-orig/elf/dl-load.c	2005-04-06 04:50:10.000000000 +0200
+++ glibc-2.3.5/elf/dl-load.c	2005-09-18 23:44:19.000000000 +0200
@@ -1143,6 +1143,10 @@
 					      c->prot,
 					      MAP_COPY|MAP_FILE|MAP_DENYWRITE,
 					      fd, c->mapoff);
+
+	/* Read the whole library sequentially first to decrease load time */
+	readahead(fd, 0, maplength);
+
 	if (__builtin_expect ((void *) l->l_map_start == MAP_FAILED, 0))
 	  {
 	  map_error: