I'm replying to this mail, but I've read all answer until today.
Thats what happens if catch with your mail after vacation ...

In message <199806260636.XAA26286@lestat.nas.nasa.gov>  Jason Thorpe wrote:
> Hi folks...
> 

Looks like the old Convex approach to me :-))
I've seen/know various HSM implementations (Convex (can't recall the product 
name only 2 customers or so), fileserv, Unitree, vertias dmapi, 
omnistore (based a vxfs), SAM-FS).

> For the HSM we're building on NetBSD/alpha, I need a couple of additional
> mode bits to indicate two separate additional file states:
> 
> 	- File has been archived

I would prefer extensible attributes here, (ala veritas or so) as they
offer much more possibilities also for other applications (DMAPI , ACL's ),
migrating parts of a file and smart appends to big files (appending
2 GB to 10 GB shouldn't result in stagein of the 10GB).
It is not that complicate ...
Also you can keep the seconary-storage information in the inode.
If you've ever repaired file2secondary-storage databases after a 
crash or a database error, you prefer this :-))
(if you use a seperate database you've to synchronise 2 databases, I still 
 have to see that realiable working yet)
On the other hand, repacking becomes a mess without a second database,
but this could be just a reverse index which can be regenerated in case
of inconsistencies.

> 
> 	- File is not resident on disk

You don't need it. ( st_blocks == 0 && st_size !=0 )  and never 
archive 'empty' files.
#define	S_ISNONRES(st)	( st->st_blocks == 0 && st->st_size !=0 &&
			    S_ISARCHIVED(st->st_mode) )

Another approuch would be to have an extra HSM-database which the kernel
can read and index the information there. It is still controlled by the 
kernel but can be read by ls,find  etc. for bulktransfers. 
It is also cheaper than extended attributes per file.

> 
> I'll also be making some changes to various system utilities, like ls(1)
> (to display these extra status bits in ls -l) and find(1) (e.g. "find all
> non-resident files").

Please make that an extra option or better hls, hfind ... else you may
screw some number of innocent scripts ...

> 
> Another option is to have a separate structure for this, returned by a
> separate call, but that is bad for us in several ways (not the least of
> which is the extra, needless system call overhead to get the additional
> information; this is a big deal, since we have file systems which have
> LARGE numbers of files in a single directory, for example).

You can immplement bulk transfers of extended attributes for stuff like find
and ls which work like an extended getdents.


Stefan
> 
> Jason R. Thorpe                                       thorpej@nas.nasa.gov
> NASA Ames Research Center                            Home: +1 408 866 1912
> NAS: M/S 258-5                                       Work: +1 650 604 0935
> Moffett Field, CA 94035                             Pager: +1 650 428 6939
> 
--
Stefan Grefen                                Tandem Computers Europe Inc.
grefen@hprc.tandem.com                       High Performance Research Center
 --- Hacking's just another word for nothing left to kludge. ---