Re: pkgsrc NetBSD 2.0_RC1/i386 bulk build results 2004-10-31

[Dan McMahill <dmcmahill%NetBSD.org@localhost>]

On Wed, Nov 03, 2004 at 12:57:39PM -0500, Todd Vierling wrote:
> On Tue, 2 Nov 2004, Krister Walfridsson wrote:
> 
> > > So the problem is that "make show-depends-dirs" does not DTRT
> > > for devel/libgail-gnome.  I have no clue why...
> >
> > So I have looked some more.
> >
> > The problem is that the dependencies from buildlink3.mk files that
> > are included from other buildlink3.mk files are not added to the
> > list returned from "make show-depends-dirs".  Is this a bug or
> > a feature?
> 
> I believe it's supposed to be a feature, because second-level bl3s are not
> considered to be "direct" dependencies.  Such dependencies do not appear in
> DEPENDS, and thus aren't in show-depends-dirs.
> 
> (If a package needs the features of the second-level dependency "directly",
> it must also directly include its bl3 file rather than relying on indirect
> inclusion.  This is part of the multi-level dependency problems I've
> outlined a few times in the recent past.)
> 
> I think what the bulk build may actually want is "show-all-depends-dirs",
> which will recurse looking for all levels of dependencies.  But why this
> problem only breaks libgail-gnome, I don't really understand.
 
no, no, no!  The way the bulk build works right now is the following:

1) visit each package and list its explicit "direct" dependencies via
'make show-depends-dirs'.  No recursion.  This is recorded in an order
which torder(1) likes.  This is the 'dependstree' file.

2) call torder(1) to find get the correct order for the build.  This
is the 'order' file.

3) call mk/bulk/tflat  to generate up and down dependencies (flattened)
for each package.  This is the 'depends' and 'supports' file.

By doing it this way, we avoid all the nasty recursion which has lots
of overhead.  For example, the memoization used by tflat is also used
in the readme generation which took the readme generation from a
several week job to under an hour!

During the build, the depends and supports files are used for the following:

- when a package breaks, the 'supports' file is consulted to see 
which all packages will break as a results and it marks them as
broken.  This avoids lots of extra pkg_add/pkg_delete steps later.

- when installing the binary dependencies for a package, the 
'depends' file is consulted.  This avoids more make recursion.

So, the place we should be looking for brokenness is in the bulk
cache files.  One possibility which I haven't checked out yet is that
the dependency tree is broken within some of these packages.  A
way I've seen that is places where the dependencies are a function of
whats already installed.  That of course is annoying because 
starting from a completely clean system and building packages will
give you something different depending on the order you pick.


-Dan

--