Bear with me, it was a long day :-)

On Wed, May 24, 2006 at 06:29:17PM -0400, Chapman Flack wrote:
> joerg@britannica.bec.de wrote:
> >The major difference in that case is that bin-install does exact
> >matching and pkg_add looks for the latest version of a package.
> 
> I think this picture is oversimplified. True, 'cd foo ; make bin-install'
> will install an exact match of foo itself (which can only be out of date
> if the pkgsrc tree is),

Given that multiple directories can create packages with the same base
name, but different version numbers, other situations are possible.

> but for prerequisites of foo, install-depends
> preserves their specified Dewey wildcards and the sub-invocations of
> bin-install pass these on to pkg_add.

I was reasonable sure that this does *not* happen. And a quick scan of
mk/ doesn't show anything to support this thesis. E.g. dewey and
wildcard patterns are *not* passed down. I think the only things that
are passed down is the expected package name (used e.g. by Python
packages) and the current PKGNAME to detect loops.

> >pkg_add wouldn't pick the best match. The only situation were this makes
> >a difference is when a newer version is available as binary package than
> >available in the tree. Often this is a broken dependency though.
> 
> Under what circumstances can a binary package be available in a newer
> version than the tree knows how to build? Only, I would argue, when the
> local tree is out of date. But the converse situation is rather common,
> during the time lag between updating of the tree and the upload of new
> binary packages. There it is important to recognize that an available
> binary package will satisfy the requirement rather than forcing a source
> build of the very newest; that's why install-depends behaves as it does.

See above or imagine what an import of php6 as php-6.0.0 would create
for the current lang/php5/buildlink3.mk.

bin-install is an exact match for dependencies, since PKGNAME_REQD or
PKGNAME are simple patterns and therefore exact matches.

> >I'd prefer to see options encoded in the binary package name instead,
> >which would fix this problem as well. It would need some extensions to
> >Dewey to match options, but that's a separate question.
> 
> That would be another workable approach, but then it should not be
> forgotten that pkg_add needs to know what combinations of options to
> look for; this information cannot come solely from packages' internal
> dependency lists, because site choices are involved; those choices are
> currently recorded in mk.conf in a rich syntax parsable by make, and
> I think it's important that they be recorded in /one/ place, and
> honored consistently.

The problem with mk.conf is exactly that it is make syntax. I don't
really want to parse it or call make to parse it, for obvious reasons.
I haven't fully thought out the binary package option handling yet, so I
can't provide a good solution yet.

> >pkg_add should *never* consult the make system. The relationship if any
> >is "source build use pkg_add", but never the other way around. This
> 
> That is the question, and I'm not sure I'm yet convinced that's the best
> answer to it. The make system is where the information is.

Let me make one important correction: where the information *can* be.
The problem is that a binary-only system might not even have make. So
pkg_add has to provide a "native" configuration anyway.

> Even if some
> of the information gets written into the package files, the make system is
> where it came from. It is recorded there in compact and sophisticated
> ways using variables, arrays, conditional, inclusion, and looping
> constructs that pkg_add does not currently duplicate and arguably should
> not.

All those are very good reasons to avoid it. I believe that a static
configuration file for pkg_add to specify what non-default behaviour is
wanted is much better, since all those flexibility is expensive and
error-prone. Just think about the useful but dangerous way to specify
options on the command line of make. It is way too easy to forget them
during the next update, which can change the functionality and
dependency tree a lot.

> I don't think a user of the system will be especially impressed by the
> philosophical purity of a separation, if the effect is a system that is
> harder to use and less well integrated. Any way you slice it, the two
> systems /are/ interdependent. They are interdependent for the inevitable
> reason that binary builds are not always available for the packages you
> need, in the versions and configurations you need, when you need them,
> so a source build fallback is necessary, and yet source-building everything
> may be prohibitively expensive, so it must be easy to have packages
> binary-installed when available in the right versions/options.  The
> philosophical decisions about how the parts of our system should or should
> not interact need to be judged by how well the resulting total system
> will perform under those inescapable demands.

This is the very same problem we currently face with buildlink3 (did read
Johnny's slide from pkgsrccon? (-:).  The authoritive source for many
information like recursive dependencies, build-time options etc. should
be the *installed* package, not the Makefile. The latter can change
easily without having any affect on the package. I don't say that the
make system should not query the pkgdb, but the other way around is wrong.

Joerg