Re: Apple ][ plus 13 sector! Problems...

[Dan Foster <usenet@evilphb.org>]

First, a disclaimer -- I came to the Apple II scene fairly late. As an
user of IIe and II+ during most of the '80s, and with lower level
programming and poking-around with the IIGS environment in 1987. By
then, all I had was ProDOS, GS/OS, etc... so I never really learned the
in-n-outs of DOS 3.x, although I did read a few interesting articles on
it over the years.

The *DEFINITIVE* work that explains how DOS works, including its RWTS
(read/write track/sector) routine and the tight timing loop involved was
in, I think, an article that Dr. Sandy Mossberg wrote for Nibble in the
'80s?

One can buy all of his Disassembly Lines articles (four volumes worth,
which were spread over many Nibble issues) for USD $24.95 from:

http://www.microsparc.com/prod02.htm

Last item on the page. I'm seriously considering doing so, myself. His
work was unparalleled and probably my favorite Apple II author because
he got really deep and dirty in inner workings of things, but wrote so
well and provided useful examples and short source code to illustrate
different things... that it was just immensely interesting and useful.

By profession, he was (not sure if still living) a medical doctor, and
apparently had reverse engineered a number of things to teach himself
how things worked, and also had a flair for writing. Nibble magazine
provided the perfect outlet for that as well as an appreciative
audience.

It was probably the clearest and most comprehensive explanation
(although David Empson's is pretty good on the hardware side) of DOS
internals I'd ever seen of all time.

As I recall, Dr. Mossberg even gave you some BASIC and assembly code to
enter, to play with various RWTS stuff to get used to it. It also
included commented dissasembled listings of DOS in key sections as well.
A lot less scary once its inner workings were very well demystified. :)

I don't have this article, unfortunately. It was from a magazine that I
never personally owned -- had only borrowed it one day to read while
bored during daily commute.

The next best thing, which I *do* have, would be a book titled _Apple(R)
Programmer's Handbook_, written by Paul Irwin and published by SAMS in
1984. It's a very nice book, even if it was in the pre-ProDOS/IIGS era
because it goes into great depth to explain hardware requirements, how
the low-level details work, and how to integrate it at the programming
level.

Well, one chapter is devoted to DOS 3.3. I'm going to presume that Mr.
Irwin will forgive me for significant quotations from his excellent book
since it's been 20 years (and if you can locate a copy -- do so!).

He uses an example of a negative-ASCII (high bit set) string 'HELLO'.
How is 'HELLO' encoded in negative ASCII in hex numbers?

$C8 $C5 $CC $CC $CF $A0

How would it be stored on a 16-sector DOS 3.3 disk? Well, it uses a
sixbit encoding scheme (as David Empson previously mentioned).

So let's convert these hex digits to their binary equivalents first for
ease of processing them further.

Hex	Binary equivalent
===	=================
$C8	11001000
$C5	11000101
$CC	11001100
$CC	11001100
$CF	11001111
$A0	10100000

Now let's combine all bytes into one *LONG* binary string and then pick
out every six bits, and rewrite it as:

110010 001100 010111 001100 110011 001100 111110 100000

That's how you go from six 8-bit bytes to eight encoded 6-bit bytes,
which he calls 'disk bytes'. (Encoded bytes stored on disk.)

Every byte must have a leading two-bits that are zero. So let's write
them out again:

Hex	Binary equivalent
===	=================
$32	00110010
$0C	00011000
$17	00010111
$0C	00001100
$33	00110011
$0C	00001100
$3E	00111110
$20	00100000

DOS's RWTS routine uses lookup tables -- one is a Read Translate Table
and one is a Write Translate Table. It's done for performance reasons,
because it doesn't have time to waste in CPU calculations in these
really tight timing loops, so it just uses lookup tables.

Using either table, you can quickly convert to/from the encoded bytes.

So it can easily translate $C8 $C5 $CC $CC $CF $A0 (8-bit data bytes) to
$32 $0C $17 $0C $33 $0C $3E $20 (6-bit disk bytes) or vice versa,
through using these translation lookup tables.

He also explains how the timing requirements was figured out:

The disk hardware spins the disk at 300 RPM (5 rev/sec), providing a
capacity of about 50,000 bits per track. This effectively translates to
accessing a 8-bit byte in 32 microseconds (us), or 1 bit every 4 us.

4 us is a very, very short period of time. DOS also has to make sure it
can fetch data (and if necessary, wait after fetching data before
starting on next bit) in exactly 4 us.

I seem to recall that Dr. Mossberg had a listing of the heart of RWTS;
the actual code that handled these reads/writes were only a few
instructions long along with a short wait and had been carefully
selected to meet 4 us per bit.

If you had inserted an additional instruction or changed one of the
existing instructions to another that took additional clock cycles,
might have royally screwed up the works. RWTS was very, very carefully
crafted, was my general impression. (Not too unlike accessing, say, the
GTIA/CTIA of the Atari during VBL updates to avoid tearing effects and
having incredibly tight timing constraints.)

Why was DOS so sensitive to timing for disk access? Mr. Irwin explains
that was because of the Apple II floppy disk hardware. (He intentionally
limits his book to only DOS 3.3 and 16-sector disks, so he doesn't cover
the 13-sector disks and their hardware.)

The disk hardware calls for a design where the drive and controller is
kept in synchronization so it can reliably read disk data bytes.

How do they do that? They encode *every* single bit of data with a clock
pulse bit.

A data bit set to '1' will have two pulses: one clock pulse and then one
pulse (signifying data bit set to '1').

A data bit set to '0' will have one pulse: one clock pulse only; absence
of a second pulse signifies data bit is set to '0'.

He calls a single clock+data bit operation 'a cell'. A cell is processed
in 4 us... but the clock pulse takes 2 us, so the data bit takes 2 us.

A disk byte is naturally, 8 bits x 4 us = 32 us.

So you *absolutely* CAN'T dawdle when processing a disk byte since 32 us
is long enough for only an handful of 6502 instructions; hence, RWTS
core routines were very, very carefully crafted by hand.

(For the write side, a byte must be supplied within 32 us; if you miss
this by waiting too long or tied up too long with other things, then it
will write out zero bits.)

So the raw data on the floppy actually has these clock pulse signals
embedded on the disk, in between each data bit.

For reads, the disk hardware looks for sync bytes to figure out where a
track starts.

Once it catches enough bits set to 1 and 0 (in such a way that this
violates the rules for data bit encoding), it knows this *HAS* to be
sync bytes, and therefore, that a track starts here.

It also knows which sector it's reading because the sector ID number is
encoded at the beginning of the sector. The sectors aren't sequential;
they're interleaved for reasons of disk hardware efficiency (with
respect to stepper motor, I think?) but RWTS has a table to map between
physical on-disk sequential vs logical sector interleaving.

It then can properly sync to clock pulses, and ignore it once read a
clock pulse, and focus on extracting the data bit that comes next.

Repeat this for each and every single data byte to be read from the
track or sector.

If you have too many 0 bits (no pulses) in a row, the hardware loses
synchronization -- which is real bad if this is a data area rather than
sync bytes.

(Sync bytes are deliberately designed to have two 0-bits in a row with a
repeated pattern of them, to make it easier for hardware to figure out
where a track starts.)

It gathers a nice collection of these data bits (minus the clock
pulses), runs it through a lookup table, and then stuffs the decoded
8-bit data bytes in a buffer and returns to the caller.

That's a simplified overview of DOS 3.3's inner workings, and there's
more to it but that's the general gist. It is rather married to
hardware-imposed timing constraints.

And as David Empson says, there are all sorts of copy protection tricks
possible. Using 'half tracks' was a common one. Technically, you could
actually double DOS 3.3 disks' capacity because the hardware *can* read
additional tracks if you turn on the right magnets at the right time.

Apple elected not to double DOS 3.3's capacity this way for ordinary
floppies because the downside to 'half tracks' is that the disk hardware
may or may not reliably read them, which makes it a dangerous
proposition.

Why not reliable? Because the 'half-tracks' required even more accuracy
and less tolerance of less-than-perfect hardware or timing.

Still, quite a few copy protected disks used this technique.

Another was to write their own bootstrap loader on disk and sometimes
either a modified DOS or their own DOS from scratch.

Another was to intentionally stuff invalid pulse sequences on floppy
disks -- your average copying tool would see them as being invalid and
skip them -- and the deprotect routine would then check to see if these
expected invalid bits were *present*. (If they weren't, the disk knew it
wasn't a faithful copy, so it'd halt or reset the machine.)

Another was to use non-standard sync bytes. Copy II Plus, for example,
had a long list of software known to use these type of tricks and what
values to use to successfully copy them, as well as having special disk
read/write routines to properly replicate the weird stuff on the copy
protected disks.

...and so on. Anyway, I first got interested in how disk hardware worked
because I wanted to back up my crumbling floppy disks, and was too poor
to afford Copy II Plus back then. :-) Had no desire to distribute it
anywhere else; solely for archival backups, which became more important
as companies pulled out of the Apple II market or went out of business.

I soon found that understanding the disk hardware along with a good
grasp of assembly language and the ROM monitor usage allowed for
unlocking the secrets of copy-protected disks to allow me to duplicate
them with success.

Here's a site with reverse engineered and fully commented DOS 3.3
functions (with the blessing of Mr. Wozniak, apparently) including RWTS
for reads and writes:

http://www.textfiles.com/apple/ANATOMY/

Looks pretty good to me, although Dr. Mossberg's was more readable for
the beginner-to-RWTS because he explained it incrementally and in chunks
rather than in one big gulp. :) Still, this site is a pretty good
reference.

I know all of this was specific to DOS 3.3 / 16-sector disks, but I
believe that if you understand how this work, then conceivably, much of
that can be applied to 13-sector disks even when taking into account
differences such as the 13-sector five-bit encoding scheme.

Not having hardware for either style disks nor experience, don't know
what tools can handle conversion. I'll leave that to the seasoned Apple
II'ers :) Worst case, can study DOS 3.3 and 3.2 RWTS, understand it,
write a program to do conversion by extracting both versions of RWTS,
etc.

As usual, I welcome correction to this or any other post; especially
since Apple DOS 3.3 is not particularly my forte. :) Spoiled by ProDOS,
GS/OS, and 3.5" drives, and the SmartPort, I'm afraid. ;)

-Dan