Re: Questions on EDASM reloc ($FE) files

["Michael J. Mahon" <mjmahon@aol.com>]

mdj wrote:

> John B. Matthews wrote:
>
>
>>Indeed, the huge variety of subtypes in LBR ($E0) can be a problem when
>>the metadata gets lost in transit.
>
>
> That's true, which is why the rest of the computing world has ignored
> filesystem metadata in preference of naming convention. I always liked
> the metadata approach, assuming that
> there was still a way to deduce a files type from it's contents, which
> unfortunately on the
> Apple II isn't the case very often. But then, the metadata in
> SOS/ProDOS is there precisely for that reason, so limited machine
> resources don't need to be consumed on validating a file.

The "precious resources" are, in fact, consumed on every file open,
by the file system.  And the integrity improvement achieved by doing
this check is far less than an (ongoing) application verification of
data stream validity, which protects against data corruption, not only
mistyping.

The only advantage for putting typing information into metadata in
principle is that it can only be modified by the file system, not by
an errant program or capricious user.

But every computer system that has implemented "external" file typing
has also found a need for utilities to allow users to re-type files, and
otherwise fiddle with the metadata--no doubt in part because most file
systems do not support metadata, and so it is easily lost in a pluralist
OS world.

The net effect is that metadata is at least as likely to be corrupted
as file contents or file name--nullifying the supposed durability
advantage of metadata.

Bottom line:  a metadata approach to file system typing is best at
protecting unsophisticated users from themselves, and has very little
value and several disadvantages for sophisticated users.

And note that the protection for unsophisticated users is only from
*themselves*, since a malicious user can easily forge metadata to cause
undesired effects when a user simply "opens" a file whose type is not
clearly visible.

> I agree that it's safe in  practice... nobody is going to be RLOADing a
> REL module that they either didn't assemble themselves or know exactly
> what it is - it's a use case that just will not occur. But, the
> computer scientist in me says, no, that's currently a unique identifier
> for EDASM REL files, and reusing it defeats it's purpose.
>
> If I had a dollar for every hour of computing time/my time I've had to
> 'waste' by resorting to temporal inference or even some more obscure
> and approximate heuristic to establish the uniqueness of some data
> entity, simply because somebody decided to reuse defunct identifiers,
> I'd be a very wealthy man.
>
> I know, I know, it really doesn't 'matter' in this case, but then, it
> mattered enough to discuss it. The benefits of doing it right mean we
> absolutely don't break anything for anyone, cc65 will have it's own
> recognisable type, and the historical record of SOS/ProDOS filetype
> assignment will be preserved. To me, it's worthwhile simply for reasons
> of nostalgia.

Well, at least that reason makes more sense than an appeal to some
"slippery slope" of file type confusion that will bring down the
Apple II world.  ;-)

Seriously, all of this is not *law*--it is simply pragmatics.
If the Apple II were a vibrant, growing platform with a long and
expanding future of software development ahead of it, I'd agree
that more anal "enforcement" of some conventions has its place.
But this isn't that world, and Java isn't going to collapse
tomorrow because someone repurposed a seldom-used Apple II file
type!

Anyone who desires to create rules to live by must understand
when and how it is appropriate to break them.  Inappropriate
application of no-longer-useful rules is the root of much
evil in the world, eh?

> While we're on nostalgia, there appears to not be a technote to
> describe the EDASM file format for REL. Somebody who knows the guts or
> has the documentation should write one and submit it.

...and might even go a small way toward making the current REL type
useful again.  ;-)

>>EDASM is one of the original wastrels. It uses the auxiliary type for
>>origin metadata, so it appears to "own" the entire REL ($FE) type. In
>>practice, EDASM declines to generate a REL file with auxiliary type in
>>the range $0000-$00FF. Although the related RLOAD tool doesn't reject
>>auxiliary types in this range, I would consider the range open.
>
>
> Oh yes, but this one pales in comparison to AppleWorks. 3(!) primary
> filetypes gone, and the aux word used to (quite cleverly) allow
> filenames to appear to contain spaces and lowercase letters. Ironic
> considering that this 'limitation' is both arbitrarily imposed, and
> very easily defeated :-)

But, in practice, it turned out about right, didn't it?  Remember,
the best engineered system is the one in which all parts fail
simultaneously, at its planned end of life (the "One Hoss Shay").

A corollary is that any systematization approaches exhaustion/saturation
as the thing it systematizes is drawing to a close.  (Of course, the art
is making sure that the systematization doesn't collapse prematurely,
bringing down the system!  ;-)

The reason for this rule is simple parsimony.  Overkill exacts a toll
in complexity and performance, and should always be avoided.

"Good enough" is *exactly* good enough.

> I'm hopeful that metadata in filesystems will make a comeback soon, and
> Apple appears to be leading the way here. Spotlight computes
> interesting metadata about files from their content and stores it as
> extended attributes, and follows established metadata standards in
> doing so. Kudos to them. Unfortunately, the other company will
> undoubtedly come up with something equivalent in functionality but
> totally incompatible with the rest of the know world.

Be careful what you wish for...

Any metadata maintained by a file system will, almost automatically,
be proprietary to that file system.  As soon as there are two file
systems, or two versions of one file system, they will support non-
identical metadata, and tranfers between them will result in metadata
loss (even in the presence of "versioning").

Actually, the best examples of useful "metadata" are not maintained by
any filesystem, but are actually embedded into structured files.  The
EXIF data in .jpg files is a good example.

And any proposed metadata format should be broadly extensible, since
we expect data to outlive all our file systems and OSs.  One such design
is XML, which can represent just about anything in a data stream.

And, again ironically, the best file system support for an "external"
metadata approach is offered by Microsoft's NTFS, which represents files
as an unlimited number of separate data streams (where Apple's data and
resource forks support only two).

> It's most amusing to be having this conversation in 2006, but it's fun.
> Good to have the dialogue and establish a consensus on the 'right' way
> to go, even if it no longer really matters.

It doesn't matter too much for the Apple II (which was my original
point), but the conceptual design of data matters a great deal for
the future.  We are slowly (culturally moderated) moving toward a
view of data as self-describing, in an extensible and relatively
platform-independent format.

External metadata maintained by a file system independently of the file
content is a move in the wrong direction, as history has already shown.

And I do agree that this discussion, while of marginal value in the
Apple II world, is of great import in the larger world.

-michael

Music synthesis for 8-bit Apple II's!
Home page:  http://members.aol.com/MJMahon/

"The wastebasket is our most important design
tool--and it is seriously underused."