Re: high sys time, very very slow builds on new 24-core system

[der Mouse <mouse%Rodents-Montreal.ORG@localhost>]

> I have a new machine with 24 2Ghz Opteron cores.  It has 32GB of RAM.

> Building with sources on a fast SSD ("preloaded" into the page cache
> before the build using tar > /dev/null) and obj, dest, and rel dirs
> on tmpfs, system builds are extraordinarily slow.

What are /tmp and /var/tmp on?

> The system takes about 20 minutes to build a netbsd-5 based source
> tree with -j24 -- about the same amount of time as an older 8-core
> Intel based system running netbsd-5 requires with -j8.

> All cores spend well over 50% time in 'sys', even when all or almost
> all are running cc1 processes.

> Does anyone have any idea what might be wrong here?

You're trying to use too recent a NetBSD? :-/

More seriously, this smells to me as though something is being
serialized, causing most cores to spend far too much time spinning
waiting for that something.  Does current still giantlock anything?  Is
your SSD reasonably performant?  (If compiler temporaries are going
onto disk, contention for the disk might be causing the sort of
serialization you see here - I think there are ordering constraints
that effectively serialize many of the steps involved when lots of
things are creating and deleting files in the same directory.  Hence
the question about /tmp and /var/tmp/.)

Maybe you need to instrument something to find out what's eating so
much system time?  Has kernel profiling support bitrotted?

/~\ The ASCII                             Mouse
\ / Ribbon Campaign
 X  Against HTML                mouse%rodents-montreal.org@localhost
/ \ Email!           7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B