Re: Guile GC behaviour

["Harvey J. Stein" <hjstein at bfr dot co dot il>]

Christian Lynbech <chl@tbit.dk> writes:

 > Even though guile calls malloc for other things than heap segments
 > (for instance in scm_makstr) this would at least be able to
 > demonstrate whether or not the growth in process size is (primarily)
 > due to heap expansion or "genuine" memory leaks elsewhere.

Thinking that scm_makstr called malloc directly reminded me of a bug
in STk where it wasn't keeping track of the amount of memory allocated
to large structures - strings, vectors & bignums, if I recall
correctly.  The effect of this was that the STk binary could blow up 
to huge sizes by doing a handful of operations which garbaged lots of
these objects, such as string appends.

So, I took a look at the code & saw that it doesn't call malloc
directly, it calls scm_must_malloc which seems to count (in
scm_mallocated) the amount of memory allocated & will gc if a certain
limit (scm_mtrigger) is surpassed.  It will increase scm_mtrigger (by
~50%) if the total malloced after gc + the new malloc is close
enough to being bigger than the trigger.

Similarly, SCM_NEWCELL pops cells off the free list & will gc when the
list is empty, and then might grow the heap (calls alloc_some_heap,
making a new heap segment 2x the size of the last).

So, it seems like there isn't a problem here, namely that aside from
problems such as with gd-stats, gc will properly trigger.

Possibilities off hand:
 1. some stray mallocs laying around - maybe everything doesn't go
    through scm_must_malloc, scm_must_realloc, or SCM_NEWCELL (in
    guile or in scwm).
 2. An accounting problem I missed in the above.
 3. Misuse of scm_must_malloc, scm_must_realloc or SCM_NEWCELL which
    ends up mallocing/reallocing stuff which isn't pointed to (in
    guile or in scwm).
 4. The fact that scm_mtrigger (and I guess the number of heaps) is
    never decreased.
 5. Memory fragmentation.

Re: 4 - One could tick the limit a few times, forcing up large limits,
followed by freeing everything.  After this, guile will allow things
to grow to these large limits each time before gc is called again.
So, if you use 30mb once, you're guaranteed to keep growing to 30mb
over & over again since guile will never gc until you hit the high
water mark.

Re: 5 - I suppose usage patterns could easily lead to not much
memory being in actual use, but the OS being unable to reuse it.

Point 5 could easily be the case with scwm (I'm thinking of lots of
decorations & pixmaps).  Printing the four values (scm_mtrigger, total
heap size, scm_mallocated & scm_cells_allocated) before & after each
gc should be enough to see.

-- 
Harvey Stein
Bloomberg LP
hjstein@bfr.co.il