Has anybody done any work profiling the kernel? Are tools available to
do this?

The reason I ask is that, in recent pissing contests here, we've seen
FreeBSD to spend about 3/5 as much time in the kernel as NetBSD for a 15
minute build of 1500 C files (something like 1:30 minutes vs
2:30). Total build times were 14:34 and 16:16 (user time was 12:10 and
12:22, the rest of the difference was in waiting for disk, I guess. I
believe the machines are similarly configured (P6-200 w/256k cache, SCSI
disk w/2940UW controller, 128MB RAM) and the cache was large enough on
the NetBSD system that it kept disk accesses down below 100k/sec during
most of the cc's (ie, all the .h's were coming from cache).

Anywho, I'd really like to see the NetBSD build time match the
FreeBSD. If there is a "cookbook" for collecting kernel profile
statistics, I'd be willing to follow it, collect data during several of
these builds, and provide the results to people competent enough to do
some hand-optimization in the kernel.

(By the way, all these timings were with NetBSD 1.2 release and FreeBSD
2.1.something (whatever was released as of December). Have there been
any speed improvements in the kernel since 1.2?