pkg_comp + SMP (tested on real hardware)

[Lars Nordlund <lars.nordlund%hem.utfors.se@localhost>]

Hello

I have finally had the opportunity to do some testing of pkg_comp[1] on
SMP gear. The machine is equipped with 2.5GiB RAM and two "Intel(R) Xeon
(TM) CPU 3.20GHz" running with hyper threading enabled. It was running
NetBSD 3.0 release and I also used the 3.0 release sets in pkg_comp.

All builds has started with empty /usr/pkgsrc/packages/All directories
to have all dependencies built in each test run. I also made sure that
all distfiles were already downloaded.

I started out by building meta-pkgs/kde3 with 4 parallel build
processes. Due to various issues, it became a longer test then I had
expected. So I only did one run. (I also wanted to test that my
mkdir-based locking scheme held up. It did.)

meta-pkgs/kde3, 4 jobs
----------------------
Total time: 5:36:00
ksh time: 20170.81 real     43677.84 user     17988.42 sys

Notes:
meta-pkgs/kde3 has an unfortunate "waist" in the dependency graph.
qt3-libs -> qt3-tools -> kdelibs3 -> kdebase3. These packages were
mainly built alone. Nearly half the time was spent building only one
package.


Next I wanted to compare between different number of parallel jobs. To
save time I needed a smaller work set. I listed the following packages
in the pkg_comp config file:

devel/subversion editors/emacs devel/check devel/cmake shells/zsh
wm/fvwm-themes net/lftp

misc pkgs, 1 job
----------------
Total time: 50:53
perl build time: 4:44
apache2 build time: 2:11
ksh time: 3054.04 real      3603.45 user      1882.48 sys

misc pkgs, 2 jobs
-----------------
Total time: 25:15
perl build time: 5:18
apache2 build time: 2:35
ksh time: 2115.49 real      3693.54 user      1929.13 sys

misc pkgs, 4 jobs
-----------------
Total time, 31:15
perl build time: 8:13
apache2 build time: 4:43
ksh time: 1875.15 real      4729.19 user      2804.02 sys

Notes:
The machine is using IDE which may explain why individual packages took
so much longer when going from 1 to 2 parallel jobs.
100% improvement on the total time when going from -j 1 to -j 2 is
probably thanks to hyper threading making process switching faster. And
in pkgsrc there are a lot of processes to switch between (buildlink3).


Final notes:
Taking advantage of SMP in pkgsrc works out quite nicely. In
combination with pkg_comp it is very convenient to start larger builds
without having to pkg_delete half the running system. I also welcome
the changes to make pkg_comp portable between operating systems, as has
been discussed in pkgsrc forums not long ago.


[1]: http://www.ludd.ltu.se/~nordlund/pkgsrc/


Many thanks to GMQ Consulting (www.gmq.se) for providing the hardware!


Best regards,
        Lars Nordlund