Re: pkgsrc scanning performance benchmarks

[Joerg Sonnenberger <joerg%bec.de@localhost>]

On Thu, Dec 01, 2016 at 07:54:27PM -0600, John Marino wrote:
> I have not looked at the cause at all, but I have some experience with
> identifying usual suspects, which are:
> - unnecessary spawning of shell commands (e.g. better alternatives exist)
> - chaining commands together with pipes  (ditto)
> - a killer: using ${MAKE} to call itself to get the value of a makefile
> variable.  As an example, even grepping for a constant is a couple of
> magnitudes faster than ${MAKE} -C ${dir} -VMYVAR

"I do not understand what is going on, but I claim that it is
unnecessary and wrong".

> I do not believe the pkgsrc framework is 28 times more complex than the
> Ports Collection framework.  It's just much more inefficient.  I know such
> statements rankle some pkgsrc devs, but numbers don't lie.

If you compare Apples and Oranges, numbers do lie. It might surprise
you, but it is a well known fact that the tree scanning i.e. as part of
the bulk build is a very time consuming component.

There have been hacks proposed in the past to replace the make extraction,
but none of the proposals actually work properly, because they disable
important functional parts. This *is* a case where pkgsrc is actually
significantly more complex than ports.

Architectionally, there are three bigger parts that slow things down as
far as the scan phase is concerned:
(1) Finding the builtins and computing the resulting versions.
(2) Reducing patterns by merging ranges.
(3) Include recursion via buildlink3.mk.

The first part could be optimized to avoid needless recomputation for a
bulk build, but it is requires figuring how a reliable caching
mechanism and reviewing the side effects of existing builtin.mk files.
AFAIK, Ports doesn't have anything like this or at least only for very
isolated items.

The second part is done with the help of some external scripts because
doing it in make internally is pretty much impossible. A single
monolithic program would be faster than the repeated pkg_admin pmatch
calls, but I don't think the total time spend on this justifies the
cost.

The last one is far more tricky. The b3.mk includes hit a number of
scalability problems in make; some of them might be fixable in the
implementation, but many are likely unavoidable without actually
introdcuing e.g. lists into the core language. The history of
mk/bsd.fast.prefs.mk should be illuminating. Ports doesn't have this
problem due to the flat dependencies. There have been discussions about
potential ways to improve the situation. One change in the past was to
improve the include guards as found by cube. See mk/termcap.buildlink3.mk
r1.7.

Joerg