Re: filesystem change monitoring

[matthew sporleder <msporleder%gmail.com@localhost>]

On Thu, Aug 13, 2015 at 5:51 AM, Emmanuel Dreyfus <manu%netbsd.org@localhost> wrote:
> On Tue, Aug 11, 2015 at 02:44:10PM +0000, Emmanuel Dreyfus wrote:
>> Do we have a smart way to monitor a whole filesystem hierarchy for
>> changes?
>
> Replying to myself: the short answer is no. kqueue will not scale to
> thousands of nodes because it needs to open a filedescriptor for each.
>
> lsyncd has been mentionned in the thread. On NetBSD, it uses kqueue
> and therefore suffers the sclaing limitation.
>
> An ongoing project to fix kqueue was mentionned:
> https://wiki.netbsd.org/projects/project/kqueue-dirnotify/
>
> That situation begs for an interm solution. Since my problem is to mirror
> a file hierarchy from an host to another, I crafter an utility called
> fsssyncd as an attempt to address the problem without lenghlty kernel
> hacking (that will be necessary, but not this week!):
> http://ftp.espci.fr/pub/syncffsd/syncffsd-0.1.tgz
>
> Basic usage:
> syncfssd /usr/local/source host:/usr/local/destination
>
> From the man page:
>> syncfssd uses rsync(1) to replicate a directory tree to another host.
>> This involves detecting changes, something NetBSD is not well prepared to
>> handle. The kqueue(2) interface can detect changes, but it needs to use a
>> file descriptor for every node monitored, and this does not scales very
>> well for huge directories trees.
>>
>> syncfssd attempts to improve the situation until kqueue(2) gets better,
>> by using FFS superblock's fs_time field, which is updated on every
>> filesystem change. This lets syncfssd detects a change immediatly if the
>> filesystem is mounted synchronous, or after a few seconds if mounted with
>> WAPBL(4) enabled.
>>
>> Once a change is detected, the relevant nodes are found by walking source
>> looking for files that changed since the previous filesystem modifica-
>> tion.  The resulting nodes are deduped (i.e.: we do not retain a node in
>> a directory if the directory itself changed), and are fed to rsync(1).
>
> This logic could be embedded into lsyncd but I am not sure such work makes
> sense. Improving kqueue seems a better, more general goal.
>
> Feedbacks on this version 0.1 are welcome.
>
> --
> Emmanuel Dreyfus
> manu%netbsd.org@localhost

Syncing dirs rsync-style is served well by unison
http://www.cis.upenn.edu/~bcpierce/unison/