[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Rescuing old 5.25" disks

Ben Rudiak-Gould wrote:

[note crossposting]
I think everyone agrees that as many old Apple II disks as possible should be imaged as soon as possible, before they degrade to the point of unreadability. And many people are imaging their disks and putting them on archives like Asimov, which is good.
But most of those images are in the dsk/do/po or nib formats. That's bad. It's bad because those formats fail to preserve information about the disks that is sometimes important, and it's *really* bad for two other reasons:
* Most of the tools that generate these formats don't do nearly enough error checking, and the formats themselves have no checksum to guard against corruption subsequent to imaging. There are probably dozens or hundreds of subtly corrupt images on Asimov, and by the time anyone notices the corruption it may be too late to reimage them. In many cases it's probably already too late. 

One of the reasons that image files don't contain internal checksums
is that they can also be "written to" by emulators, and re-computing
a checksum for an arbitrary internal change is non-trivial.

Of course, the checksum could be checked at open time and re-computed
(if changed) at file close time...
* Many disks can't be represented at all in either format. As far as I know most games have been archived only in cracked versions, and while it may be worth preserving those too, surely we shouldn't lose the originals just for lack of an appropriate format.
The reason these anemic formats dominate is obvious enough: they're by far the most widely supported, including by many old tools that will probably never support anything else. But that's entirely beside the point. The first and most important goal here is to save as much information as possible about the original disks, while that information is still available. Once the disks are imaged in *any* reasonable format, it's easy to write portable open-source tools for modern machines that can interconvert between that format and formats like dsk and nib, for compatibility with existing tools. With a good disk image format, everybody wins. 

But if you can usefully convert a "universal" format to a .dsk or .nib
image, then those images would have sufficed in the first place.

In the same way that .dsk is the default, and .nib is used only when
.dsk is insufficient, I'd propose that the universal format only be
used when both .dsk and .nib are insufficient.

If this approach is taken, then the existence of a universal form of
an image is _a priori_ evidence that it is needed for correct function.


== format ==
The canonical format needs to record "all relevant information" about the disk. In general this means bit-accurate track data (including length) for each full, half, and quarter track, inter-track synchronization information, and the status of the write-protect tab. It might be nice to have a field to hold information about the disk label also (at least text, maybe even a picture). Am I forgetting anything? Of course, the half and quarter tracks on most disks are just random noise, so it should be possible to leave them out. (The problem being figuring out when it's safe to leave them out.) In the case of ordinary unprotected 16-sector (and 13-sector?) disks, it's probably sufficient to store the decoded sectors. This does lose information, but probably not information that really needs to be saved. Again, though, it's hard to detect algorithmically that this is safe. 

Bingo.
I think that the format needs a checksum. In theory this should be the responsibility of storage and transfer protocols, but it's a sad fact of life that common protocols like FTP and HTTP POST, as implemented in practice, frequently corrupt files. An easy way to add a checksum and also keep file size down is to zip or gzip the image after the fact. That's probably a good idea in any case, but I still think the format itself needs an internal checksum. 

I agree that some older browsers and clients (and some servers ;-) are
wierdly willing to use transfer modes other than binary, but this is
(and should be) becoming an anachronism.

In the event that a text-mode transfer is done, the length of the file
is almost always altered, serving as a first-line validity check.

Aside from mode issues, I have never encountered file corruption in a
FTP/HTTP transfer that was reported successful.


Are any suitable formats already defined?
* FDI 2.0 supports bit-accurate tracks. I think it has enough support for synchronization. I think it could support half and quarter tracks by putting 96 or 192 in the TPI field. It has a comment field, but it's limited to 80 characters. Frustratingly, it doesn't have any checksums.
* 2MG supports very little right now, but could easily be extended. The main problem is that it then becomes unclear what "supporting 2MG" means. You couldn't tell from the filename whether it's a legacy 2MG or the new format. Probably it would be a good idea to invent a new file extension. 

I agree with your thinking.


== imaging tools ==
What we really need is a Disk II to USB adapter which samples and buffers the raw disk signal in much the same way as a sound card, for subsequent signal analysis on a modern PC. But that's way outside my area of expertise, so I'll talk about utilities running on a real Apple II. 

...and there's a thread on this very topic.
The canonical imaging utility needs to work, automatically and reliably, for basically any disk, protected or not. That may make it slow, but no matter how slow it is, it's much faster than cracking copy protection by hand, or, worse, trying to figure out whether an image is corrupt after the fact. More importantly, most people don't have those skills, and those people probably have disks that the people with the skills don't.
I don't think that this is terribly hard. Code for accurate bit reading already exists---in software like Copy II+ if nowhere else---and I'm sure that somebody here knows exactly how it works. The problem we're trying to solve is much easier than the one solved by Copy II+, because we don't have to write the data back to a physical disk. 

But, of course, you know that Copy II Plus is unsuccessful in copying
*many* disks without detailed parameter information tantamount to a
deprotection of the disk.

And I think that many would like to be able to re-create physical
disks from "universal" images.
I seem to recall that some bit-reading code was posted to c.e.a2 recently, but diligent searching on Google Groups has failed to turn it up. I did find this article, 

  http://groups.google.com/group/comp.emulators.apple2/msg/7ff7480ee467fed6
    Message-ID: <2tvt57$7ge@lace.Colorado.EDU>

in which Chris Redmond wrote, way back in 1994,


If you are serious about writing an emulator that can use a raw bitstream
disk image, I can write a disk reader for the Apple// that will produce
a disk image containing an exact bitstream that can be calibrated for
spiral tracks &c.
which at least suggests that it's possible to do all this without special hardware. So, did he or anyone else ever write this code? It's frustrating that nothing like this seems to exist over a decade later. 

I published some code by Stephen Thomas that could read whole tracks,
including the "nibble lengths", and the Locksmith (EDD?) track-reading
code has also been published.

The problem is not in capturing the data, but in deciding what *is*
data.

My vote is to capture *everything* and let the program that reads the
image interpret it.  If experience with emulating/running the program
demonstrates that information can be discarded without any problem,
*then* it can be discarded.  If enough is shown to be irrelevant that
the image could be represented accurately (enough) as a .nib or .dsk,
so be it.
The imaging utility also needs to be able to either save the image to a local disk (3.5" or hard), or send it over a serial connection. The former is easy. The latter is already done by ADTPro, which is an open source project. At least, it's open-source in theory, but I haven't actually found the source code for the client. The package on Sourceforge only has a dsk-format image, which contains only the binary. Assuming source code for the client exists, it seems most sensible to add an accurate-imaging extension to ADTPro rather than write a whole new application.
Unless ADTPro already does this, it should also be modified to read every block twice for verification when making a cooked image, and it should not be possible to turn this off. Yes, this will slow things down, but it's really really worth it. It could be sped up by writing an efficient whole-track reader, using code like this: 

  http://groups.google.com/group/comp.emulators.apple2/msg/fe3909783963507d
    Message-ID: <t4eoa2hg9bavhmhgpdsuic0fr5noppio54@4ax.com>

== conversion tools ==
There are lots of format-conversion utilities on Asimov, but unfortunately most of them are closed-source or system-specific or both. There needs to be a free (as in speech), portable conversion utility written in C that supports at least dsk, nib, 2mg, and the canonical format, whatever it is. I can write this from scratch if necessary. 

== direct support ==
Obviously it would be nice if some emulators supported the canonical format directly, partly because of convenience but mainly because it's the only way to support many copy-protected disks. I don't think that adding this support would be very hard. You don't need to simulate the full Woz machine, just add half track and quarter track seeking and update the data register at four-cycle intervals. 

But even that updating needs to be done in accordance with the
Woz state machine, since the reading program may be doing nibble
timing, and the Woz machine doesn't uniformly shift the register.
I'm not going to worry about emulator support for writing to the canonical format, or about converting from the canonical format back to a physical disk. Not that those wouldn't be useful features, but in the short term I'm interested mostly in archiving old disks. 

But it is vital to consider these requirements in designing a new
image "standard" for the Apple II.


== conclusion ==
This needs to be done. It should have been done a decade ago, but it's never too late. I don't think it's very hard. Comments requested. 

I agree that it should be done--but it's also not easy.  ;-)

-michael

NadaNet networking for Apple II computers!
Home page:  http://members.aol.com/MJMahon/

"The wastebasket is our most important design
tool--and it's seriously underused."