Re: Questions on EDASM reloc ($FE) files

["Michael J. Mahon" <mjmahon@aol.com>]

mdj wrote:
> Michael J. Mahon wrote:
>
>
> > The "precious resources" are, in fact, consumed on every file open,
> > by the file system.  And the integrity improvement achieved by doing
> > this check is far less than an (ongoing) application verification of
> > data stream validity, which protects against data corruption, not only
> > mistyping.
>
>
> This is only the case for the BASIC interface; ProDOS makes no
> requirement that a file type be specified when issuing an open call.
>
> The vast majority of applications fail gracefully when data stream
> validity is compromised, as they of course should. Assuming that the
> data actually valid because of it's type is obviously poor design, and
> this applies equally to metadata based filesystem and convention based
> systems.

Here we are in complete agreement...

> > The only advantage for putting typing information into metadata in
> > principle is that it can only be modified by the file system, not by
> > an errant program or capricious user.
>
>
> I disagree. The advantage of filesystem metadata is to create a single,
> well defined interface for metadata that all applications can make use
> of - ie. standardization. The system for the most part worked fairly
> well. In addition to type, the system also provides other useful
> metadata about a file, ie. the load address of a binary file, the
> record length of a text file.

I have no problem with standardization of metadata formats and
structures (as long as they remain extensible), but putting this
information outside the file, in the file system, hampers free data
interchange and begs for information loss as data is transported,
either from place to place, or from time to time, as when data
created on older systems is moved to newer systems.

The fundamental problem (if it really *is* a problem) is that there
is not now, and likely never will be, a "universal metadata standard"
which would be infinitely flexible and extensible so that it would
never become obsolete, but compact and efficient so that it would be
readily useful even on the smallest systems.  (Although there are
artful approaches to such a design, the fundamental conflict remains.)

As Maurice Wilkes once mused, in a retrospective on a groundbreaking
object-oriented computer system design, "If God had wanted things to
have unique names, He would have assigned them."  (Ironically, it may
be that He *almost* did, if we consider a hash of an organism's genome!)

> The vast majority of advantages to metadata systems are applicable to
> filesystem management applications rather than specific applications
> that create/modify data. For instance, with modern metadata systems
> such as Apple's Spotlight, or Google Desktop, the metadata that these
> applications mine from files permits much more powerful file locating
> capabilities than systems have previously had.

There is value in an open standard for attaching search (and other)
attributes to files--I just see no reason whatever for separating
that information from the contents of the file--and many reasons for
keeping it within the file, where it already largely resides.

This could, with appropriate file system support render the issue
we are discussing moot.  Consider that what we currently call a
"file", which is conceptualized as a single stream of data, becomes
a "multi-stream", with the data being only one "fork", and other
streams contain descriptive, prescriptive, or other metadata.

If such an extensible file system became the norm, with all the
metadata subsumed under the file, itself, and with a standard
way of linearizing the structure for transmission, then the
benefits of both extensibility and fixed data stream format
would be easily supported.

The Apple "forked" file system was a step in this direction, but
was on a minority platform and therefore received marginal support.
NTFS provides a generalization of this--a true multistreamed file
system--and is well on its way to pervasiveness.

If this results in a standard linearization and archive format,
and a new "minimum" standard for file systems, then we will be
much closer to achieving the goal of a uniform, extensible
implementation of "structured" files, with uniform support for
metadata.  If this is then combined with a platform-independent
extensible format for metadata, like XML, for example, then the
benefits of "attached metadata" will finally be attained.

I figure we're still at least 15-20 years from this, in a
favorable wind, but it's a worthwhile goal.

Again, note that it preserves the extensible metadata we both
want, and creates a new "standard package" for a file that
retains both its data and its metadata, independent of any
particular file system (except that all file systems must
support the multi-streamed file "package", or be locked into
dealing only with the linearization of the file).

My fundamental point:  metadata is *not* coupled to the OS or the
file system; it's coupled to the *file*.

> This system even has the
> advantage of being able to point out to a user that files are
> inappropriatly 'typed' and correct it.

This kind of fussiness and "helpfulness" by an OS is one of the
things I least like about "strong" file type systems, and the
strongest motivation for a capable "type editor"!

I am perfectly capable of managing my files and their types, and
deciding which programs I want to process which files.  I frequently
use several different programs to process the same file type for
different reasons.  I even sometimes like to open code files in a
text editor!--for good reasons.  Fixed associations are mere
convieniences, which are as frequently overridden as not.  In fact,
"adaptive" associations that learn from what I've done recently can
be even more convenient, but also more capricious and in need of
overriding.

> > But every computer system that has implemented "external" file typing
> > has also found a need for utilities to allow users to re-type files, and
> > otherwise fiddle with the metadata--no doubt in part because most file
> > systems do not support metadata, and so it is easily lost in a pluralist
> > OS world.
>
>
> This is true - but these are all side-effects of data exchange between
> hetrogenous systems, and the obvious need to defer to the lowest common
> denominator in such cases. Where there an industry-wide standard we'd
> not have that issue. I'm hopeful though that this will happen, and the
> W3C has provided a framework to do so through the 'Semantic Web'. We'll
> all benefit a great deal from the universal adoption of these
> standards.

Yes, there is a faint glow forming on the distant horizon...  ;-)

> > Bottom line:  a metadata approach to file system typing is best at
> > protecting unsophisticated users from themselves, and has very little
> > value and several disadvantages for sophisticated users.
>
>
> Well I for one am grateful that I had 15 characters to play with in a
> filename rather than 12 ;-)

Right--those were simpler times, weren't they?  ;-)

Now we all have access to filenames longer than we can view across
our wide screens, so the naming convention approach is hardly restricted
by namespace.

> > And note that the protection for unsophisticated users is only from
> > *themselves*, since a malicious user can easily forge metadata to cause
> > undesired effects when a user simply "opens" a file whose type is not
> > clearly visible.
>
>
> No. The effects of malicious forging or falsification of data apply
> equally to either system.

But the forged type is manifest in the naming convention approach,
but obscured in the metadata approach.  (I know that Windows also allows
users to supress display of know type suffixes, but using that option
is, in my opinion, a serious security risk.)  Of course, if MacOS
or other system forces the display of the meta type with a unique
and unforgeable icon, for example, then the systems are equivalent.

> > Anyone who desires to create rules to live by must understand
> > when and how it is appropriate to break them.  Inappropriate
> > application of no-longer-useful rules is the root of much
> > evil in the world, eh?
>
>
> Sure. However the usefulness of the rules is the very core of this
> debate, and the fact
> the debate even exists possibly the greatest evidence that the rules
> are still needed.

In the Apple II world, it is evident that there is no longer
any usefulness in preserving cobweb-encrusted conventions at the
expense of the meager new development.  And I think we all agree
that the liklihood of mischief from the proposed type "overloading"
is miniscule compared to the benefit for Contiki file users.

All the subsequent discussion has been based on "general principles"
rather than on expedient and reasonable pragmatics.

Count me as a "reasonable pragmatist" in Apple II typing, but as
a "realistic idealist" in file-metadata evolution for the future. ;-)

> > But, in practice, it turned out about right, didn't it?  Remember,
> > the best engineered system is the one in which all parts fail
> > simultaneously, at its planned end of life (the "One Hoss Shay").
>
>
> Yikes! This is a flawed principle, since it is not engineers who
> determine end of life, it' s users! I'll try to restrain myself a
> little and not list the endless examples of trouble that miscalculating
> a products end of life has caused, but I must at least mention Y2K...

Nonsense.  Every design decision we make contains an element of
choice related to design longevity.

Where additional longevity is inexpensive or free, we properly opt
for longevity.  But where longevity exacts a dear price for a distant
benefit, we properly opt for cost savings.

As Don Knuth remarked when asked how he thought we ever got into the
Y2K problem, "Wait until the year 10000!"

Tradeoffs are the heart and soul of every engineering activity.
"You never get something for nothing, unless earlier you got nothing
for something."  TANSTAAFL.  ;-)

> For mine, the best engineered system is the one that remains useful
> long after its end of life, and doesn't fail simply because somebody
> believed that nobody else would be using it.

Yes, I love that, too--particularly when it didn't price the
original product out of the market!

If it had turned out that commercial grade 74LS parts started to fail
after 20 years, then all Apple II's would be inoperative today.  If
this had been known at the time, and Apple had the choice of choosing
mil-spec parts, with a lifetime of 50 years, but a price of 5x the
commercial parts, choosing mil-spec parts would have been a showstopping
engineering error.  Fortunately, we've lucked out on the intrinsic
reliability of commercial grade semiconductors.  ;-)

There are a lot more C=64's than Apple II's, and it's not because
they were better engineered.  ;-)

BTW, are you aware that the longevity of spacecraft has become a
serious problem (space congestion, spectrum usage, tracking and
data collection resources), and that now spacecraft are being
designed so that they can be killed?  ;-)

> Indeed, it is only with history that the quality of any design is
> visible. A classic example from the CPU industry would be the m68k vs.
> i386. Most people would consider the 68k a superior architecture to
> i386, based primarily on the ease of assembly language programming of
> the 68k. However as both those designs matured, the two respective
> companies took markedly different approaches in moving performance
> forward. Motorola made the engineering decision that their instruction
> set architecture was no longer scalable, so teamed up with others and
> designed a new improved RISC architecture, which could get more
> performance from a supposedly simpler ISA, on the assumption that
> compilers could optimise better for said ISA. Intel on the other hand,
> said no, our ISA might be old, and might be ugly, but we'd be better
> off putting our resources into improved CPU design techniques like
> register renaming and superpipelining. The net result was that there
> wasn't much difference in performance between a Pentium and a PPC, but
> the decisions made by the Motorola team had tremendous downstream
> repercussions. Apple to their credit handled this transition
> beautifully. However, a few more years down the track the advantages
> bought along by the RISC architecture had been absorbed, and Motorola
> found themselves having to implelement the same hardware tricks as
> Intel in order to build a competitive product. G3 wasn't all that
> different than Pentium II when it came down to it.

Those fancy techniques were actually first applied in RISC machines,
because their complexities were easier to master in that enviroment,
but they were picked up by Pentium designers in a perceived fight for
survival against RISC challengers.

And CISC designer's application of advanced parallel technologies did
not deliver all that was available to RISC designers, but got them
within a factor of 1.5-2.5, which, in a world dominated by volume
economics, was enough to stem the tide and "cut off the oxygen" of
the RISC manufacturers (except in niches where code compatibility is
not more important than computational efficiency).

Put another way, register renaming works even better if the compiler
can do it.  ;-)

> And so here we are in 2006. Apple now builds machines with Intel
> processors, and even more ironically, run a variant of UNIX on them. In
> the end, which was the better engineered processor?

Measured by market success, the results are in.  Beta vs. VHS.
Innovation vs. market dominance.

"Ultimately", hardware code compatibility is not the critical issue (as
HP, Apple, and a few others have demonstrated, it can all be finessed).
But in the short run, FUD (Fear, Uncertainty, and Doubt) are powerful
forces in the marketplace, and accepting change is as difficult as
trusting that your system vendor can make change "bumpless".

> > A corollary is that any systematization approaches exhaustion/saturation
> > as the thing it systematizes is drawing to a close.  (Of course, the art
> > is making sure that the systematization doesn't collapse prematurely,
> > bringing down the system!  ;-)
>
>
> The corollary then, is a more valuable measure.

No, it's the same thing--the systematization governing a design
saturates/collapses at about the same time as the design itself,
so nothing was spent on unnecessary generality or complexity.

> > The reason for this rule is simple parsimony.  Overkill exacts a toll
> > in complexity and performance, and should always be avoided.
>
>
> I'm all for applying rules of logical reductionism. In this case, we're
> comparing changing a set of files from a generic type, to a more
> explicit type. One choice is no more complex than the other. Your
> argument (being the limited utility of filetype metadata and its
> problems)  would support the position of not changing it at all, as
> would a parsimonious argument. If we consider just the two alternative
> new types, one is ambiguous, one is not. Which one is the less complex
> choice?

You mistake my point.

I'm arguing that metadata is useful, but should be a part of the
file, accessible to applications, not maintained outside it by the
file system or "enforced" by the OS.

> > "Good enough" is *exactly* good enough.
>
>
> I do actually agree with you, when it's all said and done. But if we're
> going to change something, let's at least pick the simpler/saner
> change.
>
>
> > And any proposed metadata format should be broadly extensible, since
> > we expect data to outlive all our file systems and OSs.  One such design
> > is XML, which can represent just about anything in a data stream.
>
>
> Agreed. The proposals put forward by the W3C, built on a foundation of
> XML fulfill all these expectations. There is at this stage only limited
> use of them in the wild, but it grows, slowly but surely.

One can hope.  ;-)

> > And, again ironically, the best file system support for an "external"
> > metadata approach is offered by Microsoft's NTFS, which represents files
> > as an unlimited number of separate data streams (where Apple's data and
> > resource forks support only two).
>
>
> Hmm. UNIX systems have for years supported arbitrary metadata on files,
> which take the form of (essentially) key-value pairs. This is used by
> Apple under OS X to store the metadata mined by Spotlight, and it works
> quite well. Microsofts model is interesting, but remains unused as far
> as I know.

The fundamental issue is what happens upon transport of the files,
either on interchange media or on networks.  To be really useful, the
metadata must be inseparable from the file data, and not maintained
at the whim of the OS.

> > External metadata maintained by a file system independently of the file
> > content is a move in the wrong direction, as history has already shown.
>
>
> It's the only way metadata can go. Systems now exist that can mine data
> from image files, such as whether or not the picture contains trees, or
> how many people. We can't conceivably alter every image format to
> include such metadata, nor would it be sensible to do so.
>
> The ideal system is both extensible, and independant of existing file
> content models.

Again, I think we are in "radical agreement" on every point except the
normative role of the OS and its file system.

Let the file system remain neutral--let the metadata be a part of the
file itself.  In the best of all worlds, let the notion of a "file"
expand to support structured (multi-stream) files; until then, put
the metadata into the file whenever possible (and design new file
formats to incorporate it).

-michael

Music synthesis for 8-bit Apple II's!
Home page:  http://members.aol.com/MJMahon/

"The wastebasket is our most important design
tool--and it is seriously underused."