Re: readtable API

[Per Bothner <per at bothner dot com>]

Felix S Klock II <pnkfelix@MIT.EDU> writes:

> This is an example of what I was referring to earlier when I asked how
> throughly we were aiming to support the original Common Lisp API.  If
> we're seeking to emulate all of Common Lisp, then we should be able to
> run the examples Steele gives in his book without modification, no?

It woudl be nice, modulo converting nil->#f, t->#t, eq->eq?,
setq->set!, handling of multiple values, etc.

> According to Steele's reference, 
> 
>     "get-macro-character returns the function associated with char
>      and, as a second value, returns the non-terminating-p flag; it
>      returns nil if char does not have macro-character syntax. In each
>      case, readtable defaults to the current readtable."
> 
> In the readtable.scm source code, it seems that we ARE returning
> (invoke entry 'isNonTerminating) as a secondary value.  (Or at least
> we would be if we called 'values' instead of 'value').  That is "the
> bit whether the macro is terminating or not", no?  Or is
> 'non-terminating-p' something else for us?

Well, for Scheme, the semantics are undefined (non-portable), since
values can only be used in a call-with-values context.  However. for
Common Lisp, the semantics of multiple values are such that the extra
values are silently *ignored* except in certain contexts - and
procedure argument list is not one of those.

Kawa is a little schizophrenic about this.  For XQuery it is useful
to treat multiple values (which is how I represent XQuery sequences)
as first-class values that can be bound to variables and passes as
parameters.  Neither is possible for standard Scheme or Common Lisp.
Not sure what would be best for Kawa.

> Another question: how strongly do we want to conform to the Common
> Lisp interface in readtable.scm ?  "Extra" values aren't as easy to
> discard in Scheme as they are in Common Lisp, so perhaps finding a new
> interface, or at least adding an alternative readtable lookup
> procedure that doesn't involve 'values' would be worthwhile...

I agree.

> p.s. I don't suppose I could somehow get the changes to colon/keyword
> stuff that I was asking about earlier incorporated into all of this
> somehow?  Probably not, since Kawa wants the colon at the end of the
> token, so that has to be pretty special syntax-rewriting that is out
> of the space of readtables, no?

Well, that is handled in the token-recognition phase, not by the
readtables.  The cleanest way to change this would be to override
the handleToken method.  There is no standard CommonLisp API to
modify token parsing (i.e. the "Parsing of Numbers and Symbols"
section of CLtL:2), but that doesn't mean we can't add one.
For example:

(set-token-parser FUNCTION [READTABLE])

would change the (READTABLE so lexers using it would call FUNCTION
instead of handleToken to translate a token to a value.  The FUNCTION
abstractly takes a string as input, though creating an FString object
would hurt performance, so we might have a set-primitive-token-parser
function that takes indexes into a char[] buffer, like handleToken does.
-- 
	--Per Bothner
per@bothner.com   http://www.bothner.com/per/