Benchmark results for i386/amd64, native/Xen, TLS/noTLS

[Jean-Yves Migeon <jeanyves.migeon%free.fr@localhost>]

So, I told some people that I'd run benchmarks to qualify the TLS 
(Thread Local Storage) vs no-TLS overhead, both for Xen and native 
setups, i386 and amd64.

For results, jump directly at the bottom of this mail.

=== Context ===

To compare GENERIC and XEN3 kernels, the GENERIC kernel was always 
started UP (with boot -1). The MP support in GENERIC made it _very_ 
unfair when compared to Xen. For MP benchmarking, see the Conclusion at 
the bottom of this mail.

Host: Core i7, 24 cores, 6GiB.

i386 was always PAE enabled, both for GENERIC and XEN3. The Xen 
hypervisor used is Xen 4.1.

The no-TLS build is a release build from 2011-03-10, a few days before 
TLS support got in for x86.

The TLS build is a -currentish release build, from beginning December 
(did not take note of my last cvs up, sorry).

=== The following benchmarks were run ===

- blogbench: blogbench -d /tmp/ -i 10 -w 10 -W 10 -r 100

It (tries to) reproduce a typical blog setup, with readers, writers, and 
updaters that modify a content with text and images (mostly). The bench 
itself runs with threads, and is quite heavyweight for the kernel (you 
can run out of file descriptors very easily when launched brute-force).

It returns a "score" for read and write performance.

- sysbench: sysbench --num-threads=128 --thread-yields=1024 \
               --thread-locks=32 --test=threads run

Short description: http://sysbench.sourceforge.net/docs/#threads_mode

That bench was more focused on LWP's scheduling. I kept all the results 
(for those interested), but my main interest is the average time to 
complete the test.

- build.sh -j2 src runs

Obviously you all know what this one does. My interest is total build 
time. The same src was used all the time, but the build is native (no 
cross-compilation, no -m). obj/tools/dest/release were rm'ed each time 
before starting the build anew.

- bonnie++: bonnie++ -r 1024 -u nobody

A popular file-system and I/O benchmark. Creates, reads and writes files 
in bytes and blocks, either sequentially or randomly. Also used to 
stress hard drives.

=== Results ===

= blogbench =

http://www.netbsd.org/~jym/blogbench.results

While it uses threads to (re)produce blog-like behavior, I believe it's 
more representative of file-system and I/O performance rather than 
threads themselves, given the results.

There seems to be a trade-off between read and write performance with 
this benchmark: when one is above average, the other one is below. The 
GENERIC average perfs are a bit higher than XEN3 (~10%) though.

Other than that, there is no clear winner between TLS/no-TLS. They are 
in the same range, for both ports.

= sysbench =

http://www.netbsd.org/~jym/sysbench.results

More interestingly, sysbench stresses threads and scheduler a bit. 
There's a clear cut between TLS and no TLS systems, with TLS-enabled 
releases being slower by 15-20% (i386 and amd64) for the work completion.

Curiously, there's no real winner between XEN3 and GENERIC. i386 Xen is 
slightly faster that native (by a few percent), while amd64 Xen is 
slower than native (again by a few percent). Likely due to the pmap 
overhead, we flush mappings constantly between kernel and userland with 
amd64 (kernel runs in ring 3 for Xen amd64, just like a typical userland 
process).

= bonnie++ =

http://www.netbsd.org/~jym/bonnie.1.html
http://www.netbsd.org/~jym/bonnie.2.html
http://www.netbsd.org/~jym/bonnie.3.html

Given that bonnie is a file-system benchmark, I did not expect too much 
deviation between TLS and no-TLS. That's generally the case, except for 
sequential (and random) file creation:

with GENERIC, the cut is about to a 1:3 ratio (ouch). The release from 
2011-03-10 has a score flirting with the 9k-10k points, while the 
-currentish kernel is more like 3-3.5k. This result is 100% 
reproducible, but only with GENERIC kernels. XEN3 kernels remain 
unaffected, and TLS/noTLS releases are on par.

I am not sure that this "regression" comes from TLS (I can only express 
doubt, I have not investigated the thing technically). There has been 
lots of work in the vfs layer for the last few months, so one of these 
changes is likely to affect bonnie's results directly.

= build.sh runs =

http://www.netbsd.org/~jym/build.results

These benchmarks were run with an UP system, build.sh being invoked with 
-j2. I can't notice any real regression there, TLS -current is faster 
(3-4%) for almost every cases when compared to no-TLS releases.

Please note that all runs were made with UP kernels.

=== Conclusion ===

Except for the bonnie++ regression, overall I did not notice any real 
performance hit between a release from before TLS commit and one from 
december.

Yeah, the "TLS vs noTLS" release is a misnomer, it's rather "release 
from march vs release from december, with bits of TLS."

I am currently making another run with a release from a few days after 
the TLS commit, and subsequent kernels up to -current to investigate the 
bonnie regression.
I am taking this opportunity to rerun the benchmarks, this time with 
GENERIC + MP (and no Xen). MP makes for a real difference, with 24 cores 
the build goes down from 2h30 to a mere 25min.

(you can now start flaming me)

(BTW, the downtime between build runs allowed me to import the 
phoronix-test-suite in pkgsrc, so if people are interested in 
prototyping a performance-regression automation tool, contact me off-list).

Cheers,

--
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost