HardPressed suggestions (fwd)

[fadden@uts.amdahl.com (Andy McFadden)]

This is a message I just posted in the WestCode category in GEnie's A2 area.
I think the suggestions here reflect what most people would like to see
in a version 1.1.

-----
(Egad, this is a long one.)

Tyler: Thanks for the info.  Hope we can get the Kangaroo stuff straightened
out pretty soon... I had it installed for a while for compatibility testing
(Seven Hills gave me a copy at Apple Expo West; naturally it didn't cause any
problems on MY system... sigh), and I can see why people would miss it.

Ken:

 >May I make a few suggestions?

Always!

 >                          Suggestion:  Have an "INITIAL SETUP" option (or
 >separate program) to compress all the folders that were marked.

Yes, this would be good.  The main reason for not doing this was that we
wanted to ship before the next ice age. :-)  One possible implementation would
be to have a "preferences" item in the FExt, and then choose between "all
folders" or "marked folders".  Then you could just click on a disk and it
would only compress the stuff in the marked folders.

Another idea for a FExt enhancement is "recompress", which would uncompress
compressed files and re-compress them with the current profile (useful if you
wanted to go back and LZSS-ify some stuff).

 >     2). Include text documents in the "standard" setups.

This is a touchy one.  There are a lot of P8 TXT files, so we decided to group
it with the ProDOS 8 stuff.  On retrospect this may have been a bad idea.  In
general, I tried to err on the side of caution, reasoning that it was better
to not have the TXT files compressed by "Standard" than to have somebody get
all upset because their P8 stuff suddenly ceased to function.

Maybe I ought to u/l a "Standard + TXT" profile; there seems to be a demand
for it.  Hey, A2 guys: where's a good place to put it?  (Like, which part of
the library?)

 >               I realized that the "Standard" profile would compress very
 >few filetypes - esp. it would not compress TXT or sound files.

Sound files don't usually compress all that well, and most of them don't
follow an established filetype or file naming convention.  Admittedly I don't
have a large collection of digitized sounds, but the ones I've played with
didn't shrink much.  If you're really interested, trying playing around with
the "Diff + AHuff" profile, found in the "Goodies" folder on the HP disk.
Just copy it into the HP.Modules folder.  See also the comment about making
modules active on page 69 though.

 >     3). Change the setup so that a different compression technique can be
 >made the default for a folder.  Then I could mark my TRUETYPE folder to be
 >compressed with LZSS, my DTP folder with LZW, etc.

Somewhere around here I should still have the original "DDGS" design
specification.  I wanted to be able to configure the heck out of it.  You
could pick different kinds of compression for different types of files in
different folders... really psychotic.

It would've been a nightmare to debug, and I don't think the user interface
could have been called "usable".  The current scheme with "marked" folders and
profiles was hashed out between Rob Renstrom and me over a rather long period
of time, and is (IMHO) an excellent compromise between flexibility and ease of
use.  (Anybody out there use NuLib?  Know what -avs5ftxt/$8000 does offhand?
Be *VERY* glad Rob kept beating on me to keep it simple. :-) )

Anyway.  What you have in mind sounds pretty reasonable, and not too hard on
the eyes, though I think your next idea is the better choice:

 >     Another possibility would be to have and editer for creating Profiles.
 >Then I could specify different compression techniques for each file type.
 >Of course, this would again be difficult for the non-programmer.

I think this could be done fairly easily.  The profiles are structured so that
individual file and auxtypes are lumped into "categories", which are then
associated with compression modules.  The breakdown in appendix C is actually
divided up (more or less) according to the categories.

For example, "Generic text" (TXT) has its own category.  So to change the
compression for TXT files, all you have to do is change the list of modules
associated with that category.

A very simple profile editor would allow changing these.  A more sophisticated
editor would allow addition or deletion of categories and shuffling of
file/auxtypes between them (for example, the "All:" profiles have only one
category in them, and all the files fall into it).

(Back in the Complicated Days before Rob came up with the profile idea,
something akin to a profile editor was going to be built into the CDev.
However, you only got one "profile"; no easy switching around.  Would've been
hard to use, too.)

 >     Either method is difficult for non-programmers.  I could easily come
 >up with a "STANDARD" profile that would provide better overall disk
 >compression than the one you have.

Can you beat "All: LZSS"? :-)       (Well, sparse files don't count. :-) )

Everybody's needs are different.  A profile editor would be a good thing to
have.  If anybody feels like writing even a simple one (perhaps as a shareware
project), I'll be delighted to explain how the stuff works. (If you're
curious, the file ":HardPressed:Goodies:Standard.ASM" is the Orca/M source
code for the three standard profiles.)

 >     4). Don't compress sparse files - they don't take up much room on disk
 >anyway, and HP kept running out of disk space when trying to compress Your
 >Money Matters files.

This gets real icky when you look at HFS, because HFS doesn't support sparse
files.  HP may well be the only reasonable way to deal with sparse files on an
HFS volume.  RLE is pretty quick, and RLE+LZW should crunch down even the YMM
files.  (There's even a "Sparse" category in the standard profiles, but at the
time I didn't know of any applications that created them.)

I suppose what HP could do is compare the EOF to the block count, and if it's
more than (say) 50% or 60% sparse, don't compress it.  This would probably be
controlled by a preference.  I dislike doing this because it adds yet another
item to the list of reasons why a file may not be compressing, but so long as
it's a preference that defaults to "off" it shouldn't confuse anybody.
Shouldn't be difficult at all though.

(Ack... step 1 is to rewrite the preferences part of the CDev to hold more
stuff...)

 >     You have a GREAT product here.   I am slowly learning how to use it
 >effectively.  So far, I have not found any conflicts (even with Kangaroo),
 >or bugs.  Thanks for waiting until it was DONE.  It works just FINE!!!
 >
 >Ken Lessing

Glad you like it!


Jeff:

 >> Change the setup so that a different compression technique can be made
 >the default for a folder.

 >Again, I agree.  This would make it so that you could rearrange you HD
 >according to programs that compress better with different profiles.
 >Perhaps take a cue from another WestCode product, InWords v1.1, and let HP
 >'scan' the folders/files and suggest the best compression method.

This is a good argument in favor of the "profile per folder" scheme, but I
would contend that a profile editor will do what you want, and with less
clutter.  The other thing is that the profiles can be dropped in or removed at
any time, so they have to be identified by name and loaded when they are
needed, which would slow things down.  (The folder mark status is in a similar
bind, but they are few in number and can easily be cached in memory.)

 >        I've wound up uncompressing my boot disk one folder at a time,
 >and found that the 'biggest' improvement was gained after expanding my 7
 >meg. font folder.  Apparently, Finder re-scans the Font folder everytime
 >you return to it.  If there is a possible improvement in this area, it
 >would be greatly welcomed.

This is, to me, the One Big Flaw in HardPressed.  It takes too darn long to
scan a folder full of compressed files, especially on HFS.  The problem is
that it has to open up each file, read the first block of data, and then close
the file... it's not too bad on a RamFAST after you optimize your volume, but
when you open up a folder with 60 or 70 compressed files in it, the wait
becomes a bit severe.

(There ARE a lot of optimizations in HP.  The Orca shell likes to scan each
file twice on HFS volumes because the filenames tend to overflow its buffers,
so HP checks to see if the previous operation was a scan on the same file.  If
so, it just returns the previous copy of the info, avoiding a second
open/read/close.)

I've got some ideas on how to deal with this.  It's not hard to implement, but
making sure that extra data files are perfectly synchronized with what's
actually on the disk is critical.  It's not something I would want to ship
with the first version; too many things that could go wrong.  It'd also
require a lot of testing to validate it, and we had to stop SOMEWHERE. :-)

 >All in all, I must say that I am very impressed.  This truly is a
 >'must-have' product for all GS users.  Not one bug yet, either.
 >
 >Jeff

Thanks!  We had about 5 months of testing, by a variety of people with
discriminating tastes. :-)  BTW, if Greg Da Costa wants to beta-test something
of yours, LET HIM!!  He deserves a great deal of credit for the proper
functioning of HP.

- Andy

-----

-- 
fadden@uts.amdahl.com (Andy McFadden)
[ Above opinions are mine, Amdahl has nothing to do with them, etc, etc. ]