Re: Pbulk scan phase acceleration with tmpfs?

[Joerg Sonnenberger <joerg%bec.de@localhost>]

On Sat, Apr 10, 2021 at 10:51:49AM +0100, Jonathan Perkin wrote:
> * On 2021-04-10 at 10:37 BST, Matthias Petermann wrote:
> 
> > So what came to my mind - would it make a significant difference if I
> > keep the pkgsrc structure of the sandboxes ready in the host's tmpfs and
> > mount it to the sandbox? Then this filesystem would be served quasi from
> > RAM - of which there would also be enough available. Does anyone have
> > experience with this?
> 
> The two main things that will help scan speed are:
> 
>  * Increased concurrency through additional scan processes running
>    across multiple hosts and chroots.  One per CPU is a good start.
> 
>  * Enabling PBULK_CACHE_DIRECTORY (usually to something like /var/tmp in
> each chroot, as it must be empty prior to each run).
> 
> Any other changes usually make negligible changes to scan times, whereas
> increased concurrency will, up to a point (the fastest I managed was around
> 5 minutes for a full scan) decrease times linearly.

At least in the NetBSD case, it can help quite a bit to have a separate
copy of root in tmpfs for a set of scan processes (and avoid nullfs)
to reduce the lock contention. Essentially, watch the system time and if
it is much higher than 30%, consider checking deeper with lockstat and
friends.

Joerg