Oh, forgot some other misc ideas:

It'd be nice to split hsmd into two processes: the bookeeper, and the
file mover-inner-outer.  That should make supporting different offline
storage schemes a little easier, and also allows different methods of
storing the file bookeeping information: a simple disk file, or maybe in
huge environments you want to keep it in a real database. (I could
especially see that in NASA's case... store additional fields t do
queries on, so you can quickly find all the images taken by X probe of Y
region of planet between two times, or something.)

hsmfs should also implement the O_NONBLOCK flag. Opening a nonresident
file with O_NONBLOCK should return EWOULDBLOCK and start bringing the
file in. That way, a program that knows it's going to need data in a
bit, but needs to do some calculations first, can start getting the file
ready, and parallelize things a bit.