Re: Various size of (Project) ideas for NetBSD and pkgsrc

[Marc Espie <espie%nerim.net@localhost>]

On Wed, Oct 02, 2013 at 02:42:06AM +0200, Alistair Crooks wrote:
> Yes, IKWYM, but then you will have to specify multiple PLISTs for each
> of the different sub-packages that you build.  So now that we have a
> separate functional description of the subpackage, you'll need a
> separate DESCR, too.  And it would be better to split the monolithic
> Makefile up, too, so that it's modular for each functional
> sub-package. And, as you note later on, non-trivial changes to the
> pkgsrc mk insfrastructure.
> 
> Once you've done that, and put each functional part into its own
> directory, you have exactly the same situation that we have at the
> moment.  Minus any huffing and puffing with the pkgsrc mk
> infrastructure. So, I'm struggling to see where the benefit lies.

No, you're missing a vital part, which makes MULTI_PACKAGES useful,
which is why we went that route on OpenBSD.

You still have one single build for the software.

The split occurs during packaging.

This means that you don't need to patch to build separate thingies,
because you build it all.  Doing separate builds for each plugin is
indeed, insane.

The MULTI_PACKAGES framework is all about doing *one* single build
for the software, and *then* turn around and create several binary
subpackages during fake(stage?).

So, yes, this requires separate DESCR/PLIST for each subpackage, and
a way to provide inter-dependencies. That's one major reason why
we kept PLISTs and DESCRs in the pkg/ subdirectory, btw, because
it would become unworkable otherwise.

Our rule of thumb is as follows: if it builds as one build, and
it's not outrageous, we keep it as one build. Then we may split
the components if it makes sense during packaging.


A lot of this got automated. There was a first version of MULTI_PACKAGES
that required way too  much boiler-plate code.

These days, a few key variables are subpackage dependent for us, like
FULLPKGNAME, RUN_DEPENDS, PLIST, DESCR, COMMENT.

- we have a list of MULTI_PACKAGES, e.g.,
MULTI_PACKAGES = -main -examples -mysql -pgsql

- each of these will create a separate subpackage, with valid variables.
By default, say RUN_DEPENDS-mysql = ${RUN_DEPENDS}
if we want something different, we override.

- we have flavors that more or less correspond to options, with an
extra twist: some of those flavors are declared as PSEUDO_FLAVORS,
and they can make some subpackages disappear.

This more or less happens automatically, the result is to take
some packages out of the MULTI_PACKAGES list, and create a
BUILD_PACKAGES list which is exactly the list of stuff to be built.

That way, we can take some options *out* during the configure stage,
if we want to not build every subpackage in there.

This is especially useful for a multi-architecture pkg system, because
we can also declare stuff like
ONLY_FOR_ARCHS-mono = i386 amd64

and have some subpackages not be built on arches that don't support them,
without needing to add too much boilerplate logic to the makefile.

There are also some naming rules involved, so that we can identify unique
packages, even when end up rebuilding the same stuff for various reasons
(mostly bootstrap options).

I'll direct you to look at our x11/qt4 port, or our net/avahi port,
to have an idea about what can be done with this.

(this is all documented, see bsd.port.mk(5) in OpenBSD, there's a whole
chapter about FLAVORS AND MULTI_PACKAGES).

The bottomline is, it's all fine in theory to be patching the build process 
right and left to build different components in different directories, 
but this has two major drawbacks from an engineering point of view:
- quality suffers, because you stray further from upstream, and you multiply
the number of testcases, and you need a lot of maintenance effort to keep 
that properly running.
- from the big picture of providing binary packages, this is highly
inefficient, because you unpack, and configure each major software SEVERAL 
times, often just to build a very small piece of it.

As you're now familiar with FreeBSD, note that bapt@ is also going our
route in this, more or less. As soon as you put binary packages in the
center, and realize you don't have the sheer lemmings power of debian
to make complicated design decisions work, this becomes obvious.