Rescuing old 5.25" disks

[Ben Rudiak-Gould <br276deleteme@cam.ac.uk>]

[note crossposting]

I think everyone agrees that as many old Apple II disks as possible should 
be imaged as soon as possible, before they degrade to the point of 
unreadability. And many people are imaging their disks and putting them on 
archives like Asimov, which is good.

But most of those images are in the dsk/do/po or nib formats. That's bad. 
It's bad because those formats fail to preserve information about the disks 
that is sometimes important, and it's *really* bad for two other reasons:

  * Most of the tools that generate these formats don't do nearly enough 
error checking, and the formats themselves have no checksum to guard against 
corruption subsequent to imaging. There are probably dozens or hundreds of 
subtly corrupt images on Asimov, and by the time anyone notices the 
corruption it may be too late to reimage them. In many cases it's probably 
already too late.

  * Many disks can't be represented at all in either format. As far as I 
know most games have been archived only in cracked versions, and while it 
may be worth preserving those too, surely we shouldn't lose the originals 
just for lack of an appropriate format.

The reason these anemic formats dominate is obvious enough: they're by far 
the most widely supported, including by many old tools that will probably 
never support anything else. But that's entirely beside the point. The first 
and most important goal here is to save as much information as possible 
about the original disks, while that information is still available. Once 
the disks are imaged in *any* reasonable format, it's easy to write portable 
open-source tools for modern machines that can interconvert between that 
format and formats like dsk and nib, for compatibility with existing tools. 
With a good disk image format, everybody wins.

== format ==

The canonical format needs to record "all relevant information" about the 
disk. In general this means bit-accurate track data (including length) for 
each full, half, and quarter track, inter-track synchronization information, 
and the status of the write-protect tab. It might be nice to have a field to 
hold information about the disk label also (at least text, maybe even a 
picture). Am I forgetting anything? Of course, the half and quarter tracks 
on most disks are just random noise, so it should be possible to leave them 
out. (The problem being figuring out when it's safe to leave them out.) In 
the case of ordinary unprotected 16-sector (and 13-sector?) disks, it's 
probably sufficient to store the decoded sectors. This does lose 
information, but probably not information that really needs to be saved. 
Again, though, it's hard to detect algorithmically that this is safe.

I think that the format needs a checksum. In theory this should be the 
responsibility of storage and transfer protocols, but it's a sad fact of 
life that common protocols like FTP and HTTP POST, as implemented in 
practice, frequently corrupt files. An easy way to add a checksum and also 
keep file size down is to zip or gzip the image after the fact. That's 
probably a good idea in any case, but I still think the format itself needs 
an internal checksum.

Are any suitable formats already defined?

  * FDI 2.0 supports bit-accurate tracks. I think it has enough support for 
synchronization. I think it could support half and quarter tracks by putting 
96 or 192 in the TPI field. It has a comment field, but it's limited to 80 
characters. Frustratingly, it doesn't have any checksums.

  * 2MG supports very little right now, but could easily be extended. The 
main problem is that it then becomes unclear what "supporting 2MG" means. 
You couldn't tell from the filename whether it's a legacy 2MG or the new 
format. Probably it would be a good idea to invent a new file extension.

== imaging tools ==

What we really need is a Disk II to USB adapter which samples and buffers 
the raw disk signal in much the same way as a sound card, for subsequent 
signal analysis on a modern PC. But that's way outside my area of expertise, 
so I'll talk about utilities running on a real Apple II.

The canonical imaging utility needs to work, automatically and reliably, for 
basically any disk, protected or not. That may make it slow, but no matter 
how slow it is, it's much faster than cracking copy protection by hand, or, 
worse, trying to figure out whether an image is corrupt after the fact. More 
importantly, most people don't have those skills, and those people probably 
have disks that the people with the skills don't.

I don't think that this is terribly hard. Code for accurate bit reading 
already exists---in software like Copy II+ if nowhere else---and I'm sure 
that somebody here knows exactly how it works. The problem we're trying to 
solve is much easier than the one solved by Copy II+, because we don't have 
to write the data back to a physical disk.

I seem to recall that some bit-reading code was posted to c.e.a2 recently, 
but diligent searching on Google Groups has failed to turn it up. I did find 
this article,

  http://groups.google.com/group/comp.emulators.apple2/msg/7ff7480ee467fed6
    Message-ID: <2tvt57$7ge@lace.Colorado.EDU>

in which Chris Redmond wrote, way back in 1994,

> If you are serious about writing an emulator that can use a raw bitstream
> disk image, I can write a disk reader for the Apple// that will produce
> a disk image containing an exact bitstream that can be calibrated for
> spiral tracks &c.

which at least suggests that it's possible to do all this without special 
hardware. So, did he or anyone else ever write this code? It's frustrating 
that nothing like this seems to exist over a decade later.

The imaging utility also needs to be able to either save the image to a 
local disk (3.5" or hard), or send it over a serial connection. The former 
is easy. The latter is already done by ADTPro, which is an open source 
project. At least, it's open-source in theory, but I haven't actually found 
the source code for the client. The package on Sourceforge only has a 
dsk-format image, which contains only the binary. Assuming source code for 
the client exists, it seems most sensible to add an accurate-imaging 
extension to ADTPro rather than write a whole new application.

Unless ADTPro already does this, it should also be modified to read every 
block twice for verification when making a cooked image, and it should not 
be possible to turn this off. Yes, this will slow things down, but it's 
really really worth it. It could be sped up by writing an efficient 
whole-track reader, using code like this:

  http://groups.google.com/group/comp.emulators.apple2/msg/fe3909783963507d
    Message-ID: <t4eoa2hg9bavhmhgpdsuic0fr5noppio54@4ax.com>

== conversion tools ==

There are lots of format-conversion utilities on Asimov, but unfortunately 
most of them are closed-source or system-specific or both. There needs to be 
a free (as in speech), portable conversion utility written in C that 
supports at least dsk, nib, 2mg, and the canonical format, whatever it is. I 
can write this from scratch if necessary.

== direct support ==

Obviously it would be nice if some emulators supported the canonical format 
directly, partly because of convenience but mainly because it's the only way 
to support many copy-protected disks. I don't think that adding this support 
would be very hard. You don't need to simulate the full Woz machine, just 
add half track and quarter track seeking and update the data register at 
four-cycle intervals.

I'm not going to worry about emulator support for writing to the canonical 
format, or about converting from the canonical format back to a physical 
disk. Not that those wouldn't be useful features, but in the short term I'm 
interested mostly in archiving old disks.

== conclusion ==

This needs to be done. It should have been done a decade ago, but it's never 
too late. I don't think it's very hard. Comments requested.

-- Ben