Re: Integer Basic Tokenization

[pausch@saaf.se (Paul Schlyter)]

In article <98t1cf$r69$1@news5.svr.pol.co.uk>,
Thug <thug5@optusnet.com.au> wrote:
 
> "Paul Schlyter" <pausch@saaf.se> wrote in message
> news:98sef0$5tq$1@merope.saaf.se...
>
>> Visit my apple 2 page at:
>>
>>     http://hotel04.ausys.se/pausch/apple2
>>
>> and download my utility FID, which comes with C source.  It contains
>> de-tokenizers for Applesoft Basic, Integer Basic, and S-C Assembler
>> source files (the latter are stored as "I" type files too!).  That
>> code should help you figure out how INteger Basic programs are
>> tokenized.
> 
> Thanks Paul. Your code was very useful, and I've manged to get mine almost
> working by decoding what yours does. But it still fails sometimes, as does
> your own FID utility! The examples I've found are:
> 
> 1700 REM
> 1710 T=T+H : IF NOT ST AND NOT SP THEN 1720 : GOSUB 1780 : T=T+H : IF NOT SP
> THE
> N T=T+H1
> 1720 SP=0 : ST=SV : SV=0 : IF NOT SET(L) THEN 1750
> 
> Your code (and mine too now!) misinterprets the above as:
> 
> 1700 REM
> 1710 T=T+H : IF NOT ST AND NOT SP THEN 1720 : GOSUB 1780 : T=T+H : IF NOT SP
> THE
> N T=T+H9217-11514P=0 : ST=SV : SV=0 : IF NOT SET(L) THEN 1750
> 
> (Note line 1720 has got merged with 1720, which is really odd, because the
> EOL check should fix that, I think, but anyway...)
> 
> Another example:
> 
> 2020 X0=133*L : Y0=100 : S=-2 : X1= RND(200)+150)*(1-L)
>  : FOR I=1 TO 500 : NEXT I : X3=0
> 
> Becomes:
> 
> 2020 X-20111^EHIMEM:*L : Y-20111 POKE HIMEM: : S=-2 : X14449
> RND(200)+150)*(1-L)
>  : FOR I=1 TO 500 : NEXT I : X-20367 HIMEM:HIMEM:
> 
> So, obviously it's Variable names with trailing digits which is the problem.
> (And is itoken[0] really "HIMEM:" as well as itoken[10]? I can understand
> other tokens appearing multiple times, as it would appear to be for
> different cases/usages of the command, but this doesn't make sense for
> HIMEM.)
 
One of those HIMEM:'s is probably never used.
 
What I did in order to examine the entire token table was to poke
various bytes in an Integer Basic program line, and then LIST it and
see what appeared.  And if $00 is poked into the Integer Basic line
as a token of its own, it does get listed as HIMEM: - this might be
just an unintentional side effect since, as you correctly point out,
it doesn't make much sense to have two different HIMEM:'s.
 
Also: some of the tokens can be executed only as direct commands in
Integer Basic - trying to add them to an Integer Basic program just
yields a ***SYNTAX ERROR.  Of course one can bypass this by poking
these token bytes directly into a suitable program line.
 
 
> Anyway, the fix is easy (I think). You need to add another check to the
> code, that's all. You need to add a "InVar" boolean flag to indicate that a
> variable name is being "constrcuted".  InVar would get set whenever a
> AlphaNum character is encountered that isn't part of a REM or a String; and
> it would get reset as soon as a Token is encountered. Finally, make InVar
> another exception to the "convert the following two bytes to a number"
> routine.
 
You're absolutely right!  Thanks for finding this bug for me - I'm
going to add a fix for that to FID very soon.  I never did any extensive
testing of the Integer Basic de-tokenizer, and apparently never let it
list an Integer Basic program with embedded digits in the variable
names.
 
That bug would not have appeared if Woz had reserved a special
"Binary Integer Constant Follows" token, instead of just using ASCII
'0'-'9' with the hi bit set.
 
> I still think that it is a very weird way of encoding number too! The speed
> advantage of storing the number itself, rather than the ASCII representation
> makes sense, but why not have a "Little Endian Number follows me" token and
> leave it at that?
> 
> But then, wasn't Woz himself responsible for INTEGER BASIC? If so, there was
> probably some other fantastic optimisation he was able to perform by doing
> things this strange way, saving 5 clock cycles somewhere or something.
> <grin>
 
Woz did indeed write Integer Basic.  He had taken some classes in
compiler construction, and used that knowledge to design and
implement Integer Basic in a very short time with very modest
resources.  Some rumors I've hard claimed he wrote it over a weekend;
that's obviously an exaggreation, but within a month or two he had
implemented most of it, using only a mini-assembler (and probably lots
of notes on paper).  There has never existed any official assembler
source listing for Integer Basic, and if you disassemble and examine
the code, you'll see that it jumps here and there, as if it has
been patched a lot - which is what to be expected by code written
in that way.
 
Integer Basic was for that time one of the fastest Basic interpreters
available, probably pretty much due to it's semi-compiled nature.  Just
too bad it never got floating point - Woz was on his way with floating
point too: the set of ROM's for Integer Basic and the Monitor had
some extra space, which was used for the mini-assembler, the Sweet-16
interpreter, and some little-known floating-point routines.  But those
floating-point routines never got integrated with Woz's Basic, which
forever remained an Integer Basic.
 
Before Applesoft appeared, Integer Basic was simply called Apple Basic.
 
-- 
----------------------------------------------------------------
Paul Schlyter,  Swedish Amateur Astronomer's Society (SAAF)
Grev Turegatan 40,  S-114 38 Stockholm,  SWEDEN
e-mail:  pausch at saaf dot se   or    paul.schlyter at ausys dot se
WWW:     http://hotel04.ausys.se/pausch    http://welcome.to/pausch