Re: LTO support

[Pouya Tafti <pouya+lists.netbsd%nohup.io@localhost>]

Thank you all.  The replies have been immensely helpful.  I
have learned a lot and received very usable advice and
suggestions.
 
I tried to merge and respond to the various threads together,
which in retrospect may not have been the best idea, but here
goes.

From blymn%internode.on.net@localhost Thu Aug 12 09:30:14 2021
[snip]
> I have used both LTO 2 and LTO 4 with SCSI interfaces on
> NetBSD, they work fine (though I hit a multi-tape bug
> recently but that has been fixed)

Any chance you could elaborate on the multi-tape bug?

[snip]
> I am not 100% certain but I expect the SAS drives would work fine too.
[snip]

From he%NetBSD.org@localhost Thu Aug 12 10:39:11 2021
[snip]

From macallan1888%gmail.com@localhost Thu Aug 12 18:15:01 2021
[snip]
> Plain old SCSI or SAS is probably the least problematic solution.

Thanks for all the info on FC vs SCSI/SAS.  Looks to me like a
SAS drive would be less risky and relatively easy to find.

From brad%anduin.eldar.org@localhost Thu Aug 12 13:21:54 2021
> The general answer would be "not really"..  if you mean
> "zfs send" as the method of getting the ZFS fileset or zvol
> and if you mean native to ZFS itself.  I can't remember
> anything quite like this, but I can imagine a simple
> program existing that would take a "zfs send" on stdin and
> write to tape noticing when an EOF occurred and prompt for
> a new tape.

Thanks for the suggestions.  It also looks like amanda, which
Greg mentioned, might support this.

[snip]
> It would seem to be simpler to just use a hard drive or
> some other random block storage now.  Large drives
> (possibly larger than any LTO tape you can get) with slow
> access times which are fine for backups are pretty
> inexpensive now.

My concern about hard drives is that they seem not as hardy
(ha!) to be mailed back and forth and I'm also unsure of their
shelf life.

From gdt%lexort.com@localhost Thu Aug 12 14:30:06 2021
[snip]
> <from memory, take with grain of salt>

Thank you.  Your detailed overview has been very educational
for me.

> I used to run amanda, which runs various backup programs,
[snip]
> amanda then splits the dumps into chunks of a configured
> size, typically 1GB back then, and then puts all the chunks
> onto tapes, perhaps multiple tapes.

Would manually switching tapes be an option with amanda (I'm
tape-naive and don't want to invest in a library)?

[snip]
> Probably amanda can use "zfs send" as a dump program, or
> could be taugth to do so pretty easily.

I quickly checked and it does indeed look like there are
scripts included to support zfs snapshots and send/receive.

> These days, disks are pretty cheap, but they probably cost
> more to mail, and mailing them often isn't really good for
> them.  Think about the cost of a 4T external USB drive vs
> tapes that hold 4T.

I'm mainly concerned about the durability of data stored
offline on tape for extended periods and their relative
fragility (but maybe I'm ignorant and they're sturdier nowadays
than they were 20-30 years ago...). 

> Also consider why you are doing backups.  Most people
> should be doing backups for multiple reasons:
[snip]
> - scrambled fs due to OS bug, perhaps provoked by power
>   failure
> - scrambled fs due to hardware issues (I've seen bad RAM
>   lead to corrupted files)
> - disk failure
> - whole computer failure due to lightning, power surge,
>   etc.
> - loss of entire building, or more precisely loss of
>   primary disk and local backups at the same time
[snip]

Thanks.  The above subset are my reasons.  My NAS runs on a
decade-old used server which makes me trust the RAM and HDDs
even less.  Bugs are always a concern, as are various
possibilities to lose the primary system and local backups. 

> Another strategy is:
> 
> use disks (multiple) and write to them periodically.  store locally,
> unplugged.
[snip]
> have disks at remote sites, e.g. plugged into a RPI or apu2, and
> additionally do backups to those disks over the net

With periodic rewrites, my concern is I may discover corruption
(due to RAM, OS bug, etc.) too late after all copies have been
corrupted.

As for backups over the net, here in Germany (north of Munich),
the upload speeds I get are sadly disappointing.

> Once you start thinking like this, you probably want to
> look at backup systems that are able to do deduplication so
> that you can do backups in a way where
[snip]
> I am using bup for this, and others use borgbackup.  Surely
> there are others.

Thank you.  Will check them out!

From thierry.laronde%sfr.fr@localhost Thu Aug 12 14:45:56 2021
[snip]
> FWIW, Plan9 has/had a WORM filesystem: Write Once Read
> Many, where storage was made with deduplication of blocks,
> meaning one could have too history of files, only saving
> the differences.

I actually do have a local Plan 9 server running too but I
wouldn't quite put my faith in the infallibility of the
software. ;)  It's a lot of fun but I have had the occasional
surprise from the compiler or the system itself (and my first
attempt at using nfs to access my zfs datasets from Plan 9 was
not really a success).

From abs%absd.org@localhost Thu Aug 12 17:04:58 2021
> Another couple of possible options (Not saying "don't use
> tapes", but for reference)
> 1) Backup to external disks (either in USB case or just
> bare disks hooked up to adaptor) - depending on how they
> are formatted they can be "drop in" replacements for your
> active data, so no waiting for tapes to restore
> 2) Offsite NAS with a copy of the data

My concern with offline disks is bitrot and site loss.  With
over-the-internet backups the main problem is my lousy
bandwidth (no fibre). :(

[snip]
> When NetBSD gains native ZFS encryption I may look again at
> ZFS snapshots and possibly https://zfs.rent/ or
> https://www.rsync.net/products/zfsintro.html :)

Depending on volume those seem potentially interesting. :)

--
Pouya Tafti