This is awesome - I know it took a lot of work (your section on bisection briefly alluded to that), so thanks for doing all of this - very, very useful
Hi all,
Back in September, I wrote:
> I'm trying to run a bisection to determine why builds hosted on recent
> versions of NetBSD seem to be taking significantly more system time
> than they used to, building the same thing.
I finally have some results to report. These are from builds of the
NetBSD-8/amd64 release hosted on various versions of -current/amd64,
on a HP DL360 G7 with dual Xeon L5630 CPUs (8 cores in all). The
amount of system time taken by each build was measured using time(1).
Between a -current from September 2016 and one from October 2019, the
system time more than doubled, from 4245 seconds to 9344 seconds.
The time(1) output from the oldest and newest version was:
3930.86 real 15737.04 user 4245.26 sys
4461.47 real 16687.37 user 9344.68 sys
This means that on the recent -current, on average, roughly four of
the eight cores were executing the build tools (compilers, etc),
roughly two were executing the kernel, and the remaining two were
presumably idle.
The increase did not happen all at once but in several smaller steps
as shown in this graph:
http://www.gson.org/netbsd/bugs/system-time/graph.png
For each step, finding the commits that caused it required a separate
bisection. Each bisection took 1-2 days to run, so I have only
bisected the largest steps, those of 5 percent or more. They are
listed below in order from largest to smallest, with CVS revisions
and commit messages.
38% increase:
2018.04.04.12.59.49 maxv src/sys/arch/amd64/amd64/machdep.c 1.303
2018.04.04.12.59.49 maxv src/sys/arch/x86/include/cpu.h 1.91
2018.04.04.12.59.49 maxv src/sys/arch/x86/x86/cpu.c 1.154
2018.04.04.12.59.49 maxv src/sys/arch/x86/x86/spectre.c 1.8
Enable the SpectreV2 mitigation by default at boot time.
12% increase:
2019.03.08.20.35.10 christos src/share/mk/bsd.own.mk 1.1108
Back to using jemalloc for x86_64; all problems have been resolved.
9% increase:
2018.02.26.05.52.50 maxv src/sys/arch/amd64/conf/GENERIC 1.485
Enable SVS by default.
7% increase:
2016.12.14.15.49.35 hannken src/sys/kern/vfs_vnode.c 1.63
Change the freelists to lrulists, all vnodes are always on one
of the lists. Speeds up namei on cached vnodes by ~3 percent.
Merge "vrele_thread" into "vdrain_thread" so we have one thread
working on the lrulists. Adapt vfs_drainvnodes() to always wait
for a complete cycle of vdrain_thread().
5% increase:
2018.04.07.22.39.31 christos src/external/Makefile 1.21
2018.04.07.22.39.31 christos src/external/README 1.16
[302 more revisions by christos elided]
2018.04.07.22.39.53 christos src/external/bsd/Makefile 1.59
2018.04.07.22.41.55 christos src/doc/3RDPARTY 1.1515
2018.04.07.22.41.55 christos src/doc/CHANGES 1.2376
2018.04.08.00.52.38 mrg src/sys/arch/amd64/conf/ALL 1.85
2018.04.08.00.52.38 mrg src/sys/arch/amd64/conf/GENERIC 1.489
2018.04.08.00.52.38 mrg src/sys/arch/i386/conf/ALL 1.437
2018.04.08.00.52.38 mrg src/sys/arch/i386/conf/GENERIC 1.1177
2018.04.08.01.30.01 christos src/external/mpl/Makefile 1.1
[Too many commit messages to list here, but the following from
mrg's commit of src/sys/arch/amd64/conf/GENERIC 1.489 may
be relevant]
turn on GCC spectre v2 mitigation options.
5% increase:
2019.03.10.15.32.42 christos src/external/bsd/jemalloc/lib/Makefile.inc 1.5
turn on debugging to help find problems
5% decrease:
2019.07.23.06.31.20 martin src/external/bsd/jemalloc/lib/Makefile.inc 1.10
Disable JEMALLOC_DEBUG, it served us well, but now we want performance
back. Discussed with christos.
To summarize, most of the increase was due to Spectre and Meltdown
mitigations, which I guess is not really surprising. But the 12% net
increase from jemalloc and the 7% increase from vfs_vnode.c 1.63 seem
to call for closer investigation.
--
Andreas Gustafsson, gson%gson.org@localhost