Re: Bad news from the module/environment front

[Telford Tendys <telford@eng.uts.edu.au>]

> No, this was not what I meant. Apparently we have an architectural
> problem in the current evaluator. The way environments are handled
> generates a lot of garbage. This is not just sloppy coding in the
> righthand corner of scm_deval(), but something that looks to require a
> bona fide redesign of the single most important function in the entire
> system.

OK, if you really think that there is a good design reason for using
a different type of environment then that is something to consider,
however the issue of speed should not be the primary decision maker.
Issues such as modularity and convenience are important too.

I've been looking at ``elk'' which is a very together scheme inplementation
and available for Debian now. It includes lots of X stuff, which I
can't seem to get working <scratch head> but it also includes a nifty trick
for environments. It allows you to grab the stack frame for each closure
and that becomes an opaque object called an environment. This can be
put into a variable or whatever plus you can use it as an extra arg for
eval to force the eval to occur inside the distant closure. In essence
it is a way to open a peekhole into a closure and let that closure be
used elsewhere.

elk doesn't really have a module system based of this, but I suspect
that it would be possible. Currently it uses (require) type stuff
and an autoloader just like slib -- pretty standard really.

The nice thing is that it also has a debugger that grabs the stack
frames and allows you to walk through them, evaluate things at each
stack level, inspect variables, whatever... all by using these peekholes
into closures. I can't seem to get it to give a backtrace like guile
does but it must be possible because you can trace through each frame
by hand if you want to.

Before you ask, I have no idea how much garbage this causes or how
efficient elk is. It claims to have generational GC that's about
all I know so far.

> Telford> I often write a first draft in scheme and a third or fourth
> Telford> draft in C -- no big deal, usually the C rewrite is very fast
> Telford> because by then I know WHAT it is I'm supposed to be writing.
> 
> Which is good for smallscale scripting type of applications. But what
> if you want to write a large application that uses full CLOS and
> call/cc power? Then a rewrite in C is not a simple thing.

Two points here, firstly is guile intended to support such full
blown LISP apps? Look at emacs, it's fast but most of that speed
comes from a big library of low level C code that does the guts of the
work. The average emacs application is not real big in terms of
lines of LISP code. You can do a lot with a few lines of emacs LISP
because there is such a huge function library available to call.

Secondly, I guess that my intention is always to migrate chunks to C as
I get each chunk into a sufficiently well-defined state. I'm not so
big on object-oriented stuff. I don't mind the general philosophy but
I find the syntax too tedious and call/cc is such a huge performance
slug in guile that I keep away from that too. Surely if you use too
many fancy features, you are going to be in trouble with the compiler
anyhow.

> Telford> I think the pragmatists amongst us will admit that C
> Telford> compilers will beat scheme compilers for the foreseeable
> Telford> future.
> 
> The problem is more that current scheme compilers seems to beat the
> life out of guile (I am working on getting some numbers to support
> this claim, but haven't gotten so far yet).

I'd be interested to see the report. I'm not up on the latest scheme
compilers but it seems to me that all of them have usability limitations.
If you are saying that guile should try to keep up with other schemes
in terms of compiler technology then I agree that it should be, but I feel
the module system is long overdue and more important. Without some
some simple instructions for packaging code, people can't make a decent
release.

If I release C code, I release either a library or an application.
Either way, the rules are simple -- I give it a makefile or a configure
and I produce a chunk of code, maybe some headers.

If I want to release a guile app, there will be some C, some scheme
and a lot of system dependency. I don't know what directories to
install into, I have no way of protecting private scheme symbols, etc.

The result is that what I do release can't easily be incorporated into
other people's work, so they ignore it and write their own. There is
no global sense of organisation.

If you want to write a module for perl, you tell it your module
name and it builds a skeleton source for you to begin with. Starting
from cold, I had a compiling and installing module in about 10 minutes
just from looking at the man pages. Figuring out how to move data
in and out of my functions took about half an hour more reading
but having done that, I'm basically in a position to arbitrarily
mix perl and C code. Admittedly, they have a rather odd syntax for
writing the C interface functions, but if you follow their rules
and don't get too fancy, it works.

Moreover, I can say ``make dist'' and up comes an archive file
(with version number installed) and I can send that archive to any
perl user and they will know what to do with it immediately.

There are little ``fill in the blanks'' instructions for documentation,
if you follow those it builds a man page for you. There are lots of
tolerably simple examples, and if you can't understand one, you look
at another -- they all have the same format, so you know where you
are.

Perl syntax may be loose and arbitrary but the organisation of the
overall perl system is tight as a drum and highly meaningful.

> >> If we box ourselves in by coding only for the target audience, Guile
> >> will die a slow but certain death.
> 
> Telford> I guess that perl must be doomed to a slow but certain death.
> 
> No, because Perl5 has retargetted itself for a wider variety of
> applications. Perl5 is a general purpose object oriented programming
> language, though I do not know about speed/compilation.

But perl is further down the track than guile. At version 5 it
retargetted. Even so it's main users are sysadmins and CGI authors
which are both essentially text scanning applications. How much
numerical code gets written in perl? How much GUI code?

> But then again, perl is competing against sed and awk which are
> completely terrible programming languages.

quite true

> Guile is competing also
> against other scheme implementations such as stk, scm, rscheme and
> bigloo.

It should not be. Guile is supposed to target one particular use of
scheme which is support for script extensions for applications.
The idea is that every decent sized application will eventually want
to support user scripts and they might as well support scheme as
make up their own language. Once they support scheme they can
hang front-end processors for alternative syntax if the users feel
more comfortable with that.

Why do you say that guile is competing against scm? There is
too much difficulty embedding scm in an application, scm is
highly dependent on slib so the application that embeds scm
must also distribute slib, etc. On the other hand, would a hardcore
scm user seriously consider guile? I doubt it.

Strange that you don't mention elk in your list, maybe it is
too new. However, it has a very focused target market -- application
developers who want an easy way to tack an interpreted extension
language onto their code. This is doubly dangerous to guile
because elk has hooks into various X toolkits so the lazy coder
can knock up widgetty front ends in elk while hanging some grunty
C on the back end. Every function is documented too.

> Guile has known problems in release cycle, startup time, code
> size and library structure; if we also has poor performance, why
> should people use guile then?

Well the module system is intended to improve the library structure.
If it doesn't do that then I agree, we are better off without it.
Code size will also improve if tasks can be modularised. For example,
elk has a separate library for unix specific stuff, it is a shared
object file so if you aren't interested in unix calls, you can
ignore that completely. It also puts all the X stuff into shared
objects. OK, some systems don't have shared objects, you are facing
monolithic code and can't escape. However, you can choose which
objects you want to link to and because the code is modular, you
can throw stuff away without great hairyness.

Mind you, I haven't sifted through the low levels of elk yet,
it's C++ code and takes a bit of getting used to (like anything).

> >> But still others are doing rather well in this respect. Mark/sweep is
> >> not exactly rocket science anymore, and since (any) GC does take time,
> >> it makes it all the more important not to create more garbage than
> >> necessary.
> 
> Telford> Actually, C ...
> 
> I was not thinking about C, I was thinking about other schemes and
> lisps out there. The problem with mark/sweep is that its performance
> is proportional to the size of the heap. A copying collector is
> proportional only to the amount of the live data, a generational
> collector can do even better.

It's not as simple as that, it depends on what the data is doing.
With dead-end objects (such as strings, objects that cannot contain
references) a reference counter is unbeatable, that's why perl bases
its GC on reference count because it mostly handles strings.

With a complex web of interconnected references and lists, a mark
and sweep may be the only thing that resolves it.

On the other hand, if you have a large database then a single object
references a lot of other objects, you want to avoid doing a scan
through the entire database if you possibly can.

> Telford> In short, an interpretive language has a ``performance roof''
> Telford> -- you might as well deal with it.
> 
> Agreed, we definitely need to deal with it. Apparently, the current
> interpreter can do better, but all in all (still need to get those
> numbers) my conclusion is that for larger applications with real
> demands on performance, we need a compiler. The hobbit compiler goes
> some of the way, but still needs to embrace all of guile.

That sounds fine, don't let me discourage the use of compilers.
If a compiler gets into general use then the pressure will be off the
interpreter. I'm not saying that we should ignore the use of
compilers completely but that we shouldn't expect them to be immediately
useful because a lot of work still needs doing on that front.

In the meantime, getting the language into a state where it makes
sense and is easy to use and modular is a more pressing concern.

	- Tel