Re: Integer Basic Tokenization

["Thug" <thug5@optusnet.com.au>]

"Paul Schlyter" <pausch@saaf.se> wrote in message
news:98sef0$5tq$1@merope.saaf.se...
>
> Visit my apple 2 page at:
>
>     http://hotel04.ausys.se/pausch/apple2
>
> and download my utility FID, which comes with C source.  It contains
> de-tokenizers for Applesoft Basic, Integer Basic, and S-C Assembler
> source files (the latter are stored as "I" type files too!).  That
> code should help you figure out how INteger Basic programs are
> tokenized.

Thanks Paul. Your code was very useful, and I've manged to get mine almost
working by decoding what yours does. But it still fails sometimes, as does
your own FID utility! The examples I've found are:

1700 REM
1710 T=T+H : IF NOT ST AND NOT SP THEN 1720 : GOSUB 1780 : T=T+H : IF NOT SP
THE
N T=T+H1
1720 SP=0 : ST=SV : SV=0 : IF NOT SET(L) THEN 1750

Your code (and mine too now!) misinterprets the above as:

1700 REM
1710 T=T+H : IF NOT ST AND NOT SP THEN 1720 : GOSUB 1780 : T=T+H : IF NOT SP
THE
N T=T+H9217-11514P=0 : ST=SV : SV=0 : IF NOT SET(L) THEN 1750

(Note line 1720 has got merged with 1720, which is really odd, because the
EOL check should fix that, I think, but anyway...)

Another example:

2020 X0=133*L : Y0=100 : S=-2 : X1= RND(200)+150)*(1-L)
 : FOR I=1 TO 500 : NEXT I : X3=0

Becomes:

2020 X-20111^EHIMEM:*L : Y-20111 POKE HIMEM: : S=-2 : X14449
RND(200)+150)*(1-L)
 : FOR I=1 TO 500 : NEXT I : X-20367 HIMEM:HIMEM:

So, obviously it's Variable names with trailing digits which is the problem.
(And is itoken[0] really "HIMEM:" as well as itoken[10]? I can understand
other tokens appearing multiple times, as it would appear to be for
different cases/usages of the command, but this doesn't make sense for
HIMEM.)

Anyway, the fix is easy (I think). You need to add another check to the
code, that's all. You need to add a "InVar" boolean flag to indicate that a
variable name is being "constrcuted".  InVar would get set whenever a
AlphaNum character is encountered that isn't part of a REM or a String; and
it would get reset as soon as a Token is encountered. Finally, make InVar
another exception to the "convert the following two bytes to a number"
routine.


I still think that it is a very weird way of encoding number too! The speed
advantage of storing the number itself, rather than the ASCII representation
makes sense, but why not have a "Little Endian Number follows me" token and
leave it at that?

But then, wasn't Woz himself responsible for INTEGER BASIC? If so, there was
probably some other fantastic optimisation he was able to perform by doing
things this strange way, saving 5 clock cycles somewhere or something.
<grin>


Michael