alt Hacker News> TI-BASIC programs are stored as tokens, not text: every command, function, and variable is a token of 1 or 2 bytes. The OS detokenizes (token→display string) to show a program and tokenizes (keypress/text→token) on entry; the parser walks tokens to execute.
From my memory of using a TI-83 in the late 90s, I would not be surprised if the keypad UI injects tokens directly based on your keypress, rather than "tokenizing the text". I seem to recall, for example, you could not position the cursor in the middle of a BASIC token, and if you managed to type out the tokens it would not work; you needed to find the right menu item to inject the correct token.
Replies
Yes, to type a TI-BASIC program you have to go through the calculator menus which directly insert the tokenized input into the buffer.
The weird thing about TI-BASIC is how seemingly innocent changes in the input can cause huge performance regressions e.g. https://siraben.github.io/ti84p-re/sub-tibasic-for-paren.htm...
For(I,1,N
If 0
1
End
For(I,1,N)
If 0
1
EndI can confirm that. On the TI-83, many of the TI-BASIC tokens contained lowercase characters which couldn't be typed at all - you could only type uppercase letters on the keyboard. (There were a few lowercase letters available as tokens for special purposes, but it wasn't a full set.)
Interestingly, you could print tokens in strings - e.g. you could Disp "Disp ".
Ah makes sense. I remember a younger me trying to open .8xp files back in the day and seeing gibberish, and eventually finding the TI IDE which... felt like it had been written a long time ago (the file select dialog capped the display of file names at 8.3 I think and used ~1 and ~2 etc as "the rest of the file name").