Re: Making GC aware of overall allocation?

[Greg Harvey <Greg dot Harvey at thezone dot net>]

Lars Arvestad <arve@nada.kth.se> writes:

> 
> I don't really mind small amounts of wasted space, but I don't
> understand what the SCM_MAX_ALIGN is necessary for. To protect against 
> size = 0 problems?

This is to insure that alignment is sane for anything you'd care to
stick into the memory. There's something very similar in my gc patches
if it's compiled with SCM_TRACK_ALLOCS (though it won't use that
method if the malloc provides size information).
 
> >> Miroslav Silovic writes:
> MS> Use scm_must_malloc instead of malloc in your C code. Make sure that
> MS> the free function for your smobs returns the accurate value for the
> MS> memory freed (i.e. matching the ammount allocated with
> MS> scm_must_malloc), otherwise your gc will get very confused.
> >> 
> >> But this seems to be the wrong approach IMHO. If I get my hands on a
> >> library that I want to access from Guile, I should not have to rewrite 
> >> important parts of it.
> 
> ML> well, in that case you want a plug-compatible malloc replacement to
> ML> do the bookkeeping.
> 
> ML> I don't think you can have smart things happenning in the allocator
> ML> without touching the allocator.
> 
> ML> or do I misunderstand the problem?
> 
> A plug-compatible malloc would be great, but how do I make that
> happen? Should I try to link my lib with the malloc replacement before 
> linking it with guile? I guess I could take a look at Electric Fence
> (great debug tool, btw) for example code.
> 
> My wishful thinking has been that the GC could look at the heap size
> once in a while (waving my hands here). If the heap was large and/or had
> grown more than what could be explained from Guile-side growth, a GC
> could be tried in order to release memory (yet more waving). I don't
> know Unix and Guile well enough to say wether this is possible or not.

This depends; if you have a limit set for the maximum size of a
process, it should be pretty easy to find it out using getrlimit and
stuff (and it's also possible in other ways using glibc, but it's not
portable, alas). If not (for those of us silly enough to let our
processes grow like mad until they dump core, eating up every last bit
of disk space ;), you'd have to set it by hand and make sure that all
the space (or a good approximation of the memory allocated ) is
reported to the gc through scm_done_malloc.

Still, this is only a partial solution; guile's gc currently has the
bizarre notion that there's always space available (a very foolish
thing for a gc to assume), so things like mtrigger only really help
once, then they get bumped up to a larger size. Revamping the garbage
collector's concept of when and why to collect is on the list, but it
comes after implementing the generational gc (which is so overdue it's
making Godot look quite punctual).

> What I tried to say was that (1) I don't want my lib to be Guile
> dependent, and (2) that if I use Guile functions in my library code
> anyway, I could provide replacements (for e.g. scm_must_malloc) but
> that is a bad idea because the non-Guile programmers (and myself by
> mistake) will use the good old malloc, thus introducing bugs. My point
> is basically that a general library should be allowed to be unaware of 
> libguile. 

I think that what you want to do is have any special gc stuff done in
a wrapper for guile. After all, you aren't going to just call the
library directly from guile without it (particularly if you don't want
the library dependent on guile). Currently, this probably means
fudging with scm_done_malloc; hopefully it'll become somewhat more
sane in the near future.

-- 
Greg