pkg/34974: pkgsrc sometimes compiles packages it doesn't need to compile

[kre%munnari.OZ.AU@localhost]

        Note: There was a bad value `' for the field `Class'.
        It was set to the default value of `sw-bug'.

>Number:         34974
>Category:       pkg
>Synopsis:       pkgsrc sometimes compiles packages it doesn't need to compile
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Nov 02 13:00:00 +0000 2006
>Originator:     Robert Elz
>Release:        NetBSD 3.99.15 (pkgsrc current today)
>Organization:
        Prince of Songkla University
>Environment:
System: NetBSD jade.coe.psu.ac.th 3.99.15 NetBSD 3.99.15 
(GENERIC-1.696-20060125) #8: Wed Jan 25 04:59:39 ICT 2006 
kre%jade.coe.psu.ac.th@localhost:/usr/obj/current/kernels/JADE_ASUS i386
Architecture: i386
Machine: i386
>Description:
        When using the "bin-install" pkgsrc target, and a reasonably complete
        set of binary packages (so only packages that have been updated should
        ever need to be rebuilt), pkgsrc sometimes recompiles and rebuilds a
        package that does not need to be recompiled 9that is, for which a
        binary package already exists, and which is unchanged).

>How-To-Repeat:
        I have seen this happen many times over the pasy year(s) - ie: this
        is nothing new, I'm just getting around to remorting it as I noticed
        it again today in a reasonably simple case.

        To make this happen, remove (move) binary packages (if any) for
        security/opencdk and security/wireshark - but have an existing
        package for security/gnutls.

        if you don't have the binary packages already, this is a hard state
        to arrive in, as gnutls depends upon opencdk.

        It happened for me today, as opencdk was updated a day or two ago,
        so my old binary package was removed, and wireshard was updates
        today (so its binary package was rmeoved), but gnutls hasn't been
        updated for a month and a half now.

        Then build (using the bin-install target) security/wireshark.

        pkgsrc notices the dependency upon gnutls, and attempts to install the
        bnary package (which is as it should).  The pkg_install fails, as
        gnutls requires opencdk, and there is no binary package available for
        that one (yet).   At this stage pkgsrc falls back to a source build,
        and attempts the source build of gnutls - that again needs opencdk
        as a dependency, but a source build of that succeeds without problems.

        At this point, a binary install of gnutls would work just fine, but
        pkgsrc has already abandoned that attempt (tried it, it failed...)
        So goes ahead and rebuilds gnutls from source, making a new binary
        package (which overwrites the one that was there, which is really not
        a very nice thing to happen - and in some cases could be a real problem)

>Fix:
        I suspect that the only way to fix this involves a rework of the way
        that pkgsrc handles dependencies - instead of doing recursive builds,
        it really needs to be iterative - that is, pkgsrc needs to find what
        dependencies need to exist for the package being built (the one on the
        original user command line), and then for all dependencies of that
        package, hen combine that list into a sensible form (if A requires
        C>=2 and B reqiores C>=3 and we need both A and B, then we need
        C>=3 (the C>=2 just gets absorbed).  Once we have that list, it needs
        to be sorrted into depenndcncy order, after which the packages can be
        built (whether that's a bin-install build, or just install, or
        whaveer other has been requested) in the order that means no
        (further) recursion happens - everything already exists when it is
        needed.  With a scheme like this, the build of wireshark would have
        discovered that opencdk 9as well as gnutls) was needed, and that as
        opencdk is needed by gnutls, opencdk would be built first.   There
        the pkg_install fails, as there is (was) no binary packate, and a
        source build is done.  Then gnutls comes next, now the binary install
        succeeds, as all dependencies are already installed, and n new
        compilation is required.  After that, the compilation of wireshark
        can be successfully accomplished.

        NOTE: the answer here is **NOT** "gnutls really needsd to be
        recompiled anyway so it could get correct dependency info in the
        binary package" - or if that is the answer, then that just exposes
        a similar (but much harder to fix, and much more stupid) but in
        pkgsrc - as if instead of simply building wireshark, I had first
        built opencdk (which I might have, and would have, had I gotten around
        to updating my binary package collection a day earlier), then the
        bvnary-install of the (old) gnutls package would have succeeded (it
        might have issued a strange error-looking type of message about the
        correct vesion of the binary package not being found - but it would
        have successfully installed from the binary package - and would not
        have been rebuilt.   That's inconsistent with the (assumed here) view
        that the binary package needed to be rebuilt.

        Also note tha the 3 packages opencdk, gnutls, and wireshark (and
        whetaver other dependencies they have which aren't relevant here)
        are not the point f this PR - they just happened to form today's
        example, so nothing specific to any of thse packages can possibly be
        relevant to this PR (ie: bumping PKGREVISION of gnutls would be
        a truly stupid response to this PR, whether or not it might happen
        to be called for because of some other reasons).

        pkgsrc probably needs a major reweite, and that should probably be
        done in sh, rather than in make - it uses almost none of the
        true facilities of make, and continuing to use it just because once,
        many years ago, when the project started (as FreeBSD ports) make
        seemed like the appropriate tool, does not cause that to continue
        to be true...   As a set of shell scripts I suspect pkgsrc could be
        many times more flexible, faster, and MUCH simpler to maintain.