Re: high sys time, very very slow builds on new 24-core system

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Wed, Mar 23, 2011 at 05:24:12PM -0400, Thor Lancelot Simon wrote:
> 
> All cores spend well over 50% time in 'sys', even when all or almost
> all are running cc1 processes.  The kernel is amd64 -current GENERIC
> from about 1 week ago -- no DIAGNOSTIC, DEBUG, KMEMSTATS, LOCKDEBUG,
> etc.

I am getting some lockstat output now; I will send it here as soon as
I have it.

While I wait, I can see two very curious things in systat vmstat:

        1) There is a truly huge number of zfod pages, sometimes well
           over 100,000/sec, even with 10-15% CPU idle.  I wonder if the
           idle loop page zeroer is for some reason not working efficiently
           on this system.

        2) About 200 xfers/sec (about 3MB/sec worth) is still going on
           to the SSD for much of the build process.  It's all writes.
           All build directories (obj, dest, tools, release) are on
           a separate tmpfs, /tmp is on tmpfs, the machine isn't
           using any swap, and I am building with -U -O.  By the time I
           verified all this, the disk writes were done so I couldn't
           use fstat to see what files might be being written.

           I wonder if some tool has reverted to using /var/tmp for its
           intermediate files, or similar.  Don't see an obvious
           culprit for that, however.

Thor