Re: Questions on EDASM reloc ($FE) files

["mdj" <mdj.mdj@gmail.com>]

Michael J. Mahon wrote:

> The "precious resources" are, in fact, consumed on every file open,
> by the file system.  And the integrity improvement achieved by doing
> this check is far less than an (ongoing) application verification of
> data stream validity, which protects against data corruption, not only
> mistyping.

This is only the case for the BASIC interface; ProDOS makes no
requirement that a file type be specified when issuing an open call.

The vast majority of applications fail gracefully when data stream
validity is compromised, as they of course should. Assuming that the
data actually valid because of it's type is obviously poor design, and
this applies equally to metadata based filesystem and convention based
systems.

> The only advantage for putting typing information into metadata in
> principle is that it can only be modified by the file system, not by
> an errant program or capricious user.

I disagree. The advantage of filesystem metadata is to create a single,
well defined interface for metadata that all applications can make use
of - ie. standardization. The system for the most part worked fairly
well. In addition to type, the system also provides other useful
metadata about a file, ie. the load address of a binary file, the
record length of a text file.

The vast majority of advantages to metadata systems are applicable to
filesystem management applications rather than specific applications
that create/modify data. For instance, with modern metadata systems
such as Apple's Spotlight, or Google Desktop, the metadata that these
applications mine from files permits much more powerful file locating
capabilities than systems have previously had. This system even has the
advantage of being able to point out to a user that files are
inappropriatly 'typed' and correct it.

> But every computer system that has implemented "external" file typing
> has also found a need for utilities to allow users to re-type files, and
> otherwise fiddle with the metadata--no doubt in part because most file
> systems do not support metadata, and so it is easily lost in a pluralist
> OS world.

This is true - but these are all side-effects of data exchange between
hetrogenous systems, and the obvious need to defer to the lowest common
denominator in such cases. Where there an industry-wide standard we'd
not have that issue. I'm hopeful though that this will happen, and the
W3C has provided a framework to do so through the 'Semantic Web'. We'll
all benefit a great deal from the universal adoption of these
standards.

> Bottom line:  a metadata approach to file system typing is best at
> protecting unsophisticated users from themselves, and has very little
> value and several disadvantages for sophisticated users.

Well I for one am grateful that I had 15 characters to play with in a
filename rather than 12 ;-)

> And note that the protection for unsophisticated users is only from
> *themselves*, since a malicious user can easily forge metadata to cause
> undesired effects when a user simply "opens" a file whose type is not
> clearly visible.

No. The effects of malicious forging or falsification of data apply
equally to either system.

> Anyone who desires to create rules to live by must understand
> when and how it is appropriate to break them.  Inappropriate
> application of no-longer-useful rules is the root of much
> evil in the world, eh?

Sure. However the usefulness of the rules is the very core of this
debate, and the fact
the debate even exists possibly the greatest evidence that the rules
are still needed.

> But, in practice, it turned out about right, didn't it?  Remember,
> the best engineered system is the one in which all parts fail
> simultaneously, at its planned end of life (the "One Hoss Shay").

Yikes! This is a flawed principle, since it is not engineers who
determine end of life, it' s users! I'll try to restrain myself a
little and not list the endless examples of trouble that miscalculating
a products end of life has caused, but I must at least mention Y2K...

For mine, the best engineered system is the one that remains useful
long after its end of life, and doesn't fail simply because somebody
believed that nobody else would be using it.

Indeed, it is only with history that the quality of any design is
visible. A classic example from the CPU industry would be the m68k vs.
i386. Most people would consider the 68k a superior architecture to
i386, based primarily on the ease of assembly language programming of
the 68k. However as both those designs matured, the two respective
companies took markedly different approaches in moving performance
forward. Motorola made the engineering decision that their instruction
set architecture was no longer scalable, so teamed up with others and
designed a new improved RISC architecture, which could get more
performance from a supposedly simpler ISA, on the assumption that
compilers could optimise better for said ISA. Intel on the other hand,
said no, our ISA might be old, and might be ugly, but we'd be better
off putting our resources into improved CPU design techniques like
register renaming and superpipelining. The net result was that there
wasn't much difference in performance between a Pentium and a PPC, but
the decisions made by the Motorola team had tremendous downstream
repercussions. Apple to their credit handled this transition
beautifully. However, a few more years down the track the advantages
bought along by the RISC architecture had been absorbed, and Motorola
found themselves having to implelement the same hardware tricks as
Intel in order to build a competitive product. G3 wasn't all that
different than Pentium II when it came down to it.

And so here we are in 2006. Apple now builds machines with Intel
processors, and even more ironically, run a variant of UNIX on them. In
the end, which was the better engineered processor?

> A corollary is that any systematization approaches exhaustion/saturation
> as the thing it systematizes is drawing to a close.  (Of course, the art
> is making sure that the systematization doesn't collapse prematurely,
> bringing down the system!  ;-)

The corollary then, is a more valuable measure.

> The reason for this rule is simple parsimony.  Overkill exacts a toll
> in complexity and performance, and should always be avoided.

I'm all for applying rules of logical reductionism. In this case, we're
comparing changing a set of files from a generic type, to a more
explicit type. One choice is no more complex than the other. Your
argument (being the limited utility of filetype metadata and its
problems)  would support the position of not changing it at all, as
would a parsimonious argument. If we consider just the two alternative
new types, one is ambiguous, one is not. Which one is the less complex
choice?

> "Good enough" is *exactly* good enough.

I do actually agree with you, when it's all said and done. But if we're
going to change something, let's at least pick the simpler/saner
change.

> And any proposed metadata format should be broadly extensible, since
> we expect data to outlive all our file systems and OSs.  One such design
> is XML, which can represent just about anything in a data stream.

Agreed. The proposals put forward by the W3C, built on a foundation of
XML fulfill all these expectations. There is at this stage only limited
use of them in the wild, but it grows, slowly but surely.

> And, again ironically, the best file system support for an "external"
> metadata approach is offered by Microsoft's NTFS, which represents files
> as an unlimited number of separate data streams (where Apple's data and
> resource forks support only two).

Hmm. UNIX systems have for years supported arbitrary metadata on files,
which take the form of (essentially) key-value pairs. This is used by
Apple under OS X to store the metadata mined by Spotlight, and it works
quite well. Microsofts model is interesting, but remains unused as far
as I know.

> External metadata maintained by a file system independently of the file
> content is a move in the wrong direction, as history has already shown.

It's the only way metadata can go. Systems now exist that can mine data
from image files, such as whether or not the picture contains trees, or
how many people. We can't conceivably alter every image format to
include such metadata, nor would it be sensible to do so.

The ideal system is both extensible, and independant of existing file
content models.

--
Matt