Re: fsync, rdiff-backup, wapbl, and WD Elements 1T drive

[David Holland <dholland-tech%netbsd.org@localhost>]

On Sat, Oct 29, 2011 at 07:54:44AM +0200, Alan Barrett wrote:
 > The current state of ffs+wabl is that, if the system crashes and
 > the log is replayed, then files that had been written shortly
 > before the crash end up with whatever old data happened to be in
 > the underlying disk blocks, but new metadata indicating that the
 > size and timestamps are all up to date.  I think that this violates
 > traditional unix file system semantics, but the people who worked
 > on wapbl don't seem to think it's a problem.

It doesn't violate traditional semantics because ffs has always had
this bug, but it is a bug. However, it's gotten more noticeable; I
think this is mostly because wapbl is much faster creating files, so
bulk copies/untars can accumulate much more unflushed data in the same
amount of time. Also it gets worse as memory sizes increase.

fsyncing everything is inherently expensive. fsyncing every file
(especially if many of them are small) is inherently *very* expensive.

Try hacking it to fsync in batches, or even (if the machine isn't
doing much else at the same time) to just call sync every half second
or something. This should make it markedly faster without much loss of
integrity.

However, a tool that really supports commit/abort semantics (unlike
rsync) shouldn't need to sync at all until it's done.

-- 
David A. Holland
dholland%netbsd.org@localhost