This is the mail archive of the guile@cygnus.com mailing list for the guile project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]

Tuning gc and heap allocation


I have had some problems with the heap allocation. The problem occurs when
I'm using large data structures where the total size comes close to the size
of the physical memory of the machine.

In one particular case I was running on a machine with 128Mb physical memory
and the data analysis I was doing required about 70-90 Mb of memory.

I could, however not do much. At some stage in the code the memory used by
the process rised to about 190 Mb and after that the only thing the process
could do was paging.

In an attempt to solve this we put (gc) in several places where large blocks
of memory was expected to be allocated. It helped!

A quick fix to this problem was obtained by changing the following statement
in gc.c

#define SCM_EXPHEAP(scm_heap_size) (scm_heap_size*2)

changed to:

#define SCM_EXPHEAP(scm_heap_size) (scm_heap_size/2)

I do, however, not have any clear idea how this heap allocation works. It
is obvious that it is efficient to use an exponential growth of the heap
allocation size but it is very dangerous to double the heap size every time.

A nice compromise would probably be to use 1.5 instead of 2.

I would also suggest that we should be able to put an almost hard upper limit
(as a parameter?) on the maximum allowed allocation size. As long as we don't
have generational, copying gc it is very painful when the memory size of the
guile process exceeds the physical memory size.

	Best regards
	Roland Orre
------------------------------+---------------------+-----------------
Roland Orre                   | O---O---O Studies of| orre@nada.kth.se
SANS, NADA, KTH               | |\ /|\ /  Artificial| 
S-100 44 Stockholm, Sweden    | O-O-O-O   Neural    |Wph:+46 8 7906984
------------------------------+ |/ \ /|   Systems   |Fax:+46 8 7900930
Dept. of Computing Science    | O---O-O  +---------|Mob:+46 70 8269748
Royal Institute of Technology |          |http://www.nada.kth.se/~orre
------------------------------+----------+----------------------------