> On Fri, 26 Jul 2002, David Burgess wrote:
>
>> > On Fri, 19 Jul 2002, David Burgess wrote:
>> >
>> > Issues that are at all controversial don't belong in a PR -- they
>> > need to be discussed in tech-pkg first.
>>
>> Sorry - I got the order wrong.  I thought we documented our
>> requirements and then argued about implementation.
>
> I don't believe your requirements are yet clear...

I think they are - the implementation of the requirements is a black as
tar, but the requirement certainly is clear to me.
>
>> 3)  Include a mechanism that overwrites the /usr/src source tree from
>> the installation script.  This way, the /usr/src tree agrees with the
>> /usr/* directories.  This is my least favorite, but is conceptually
>> the best, since the fidelity of the installed software base is based
>> on the source tree, not two competing sources.
>
> So, you want "/usr/src" to build and match the running system? How
> about "cvs update", you want that to work, too? With this request,
> you've stumbled into a reason why using "pkgsrc" to maintain the base
> system is so bad...

No.  This is a very poor suggestion proffered for the small but typically
vocal group that screams about "source purity" and hates immutable
configuration management baselines.
I didn't "stumble" into that reason.  I'll speak to this more in a minute.

>
>> Having to remember that /usr/pkg/sbin/makemap is the
>> way to make a new sendmail database on half of the servers, and
>> /usr/sbin/makemap works on the rest is killing me, especially when
>> "makemap" is still on the system and first in the path.
>
> Just curious here: what's the downside of using the old "makemap". Has
> the format of the hashed ".db" files actually changed?

yes it has.  They two formats are incompatible.  They will likely be
incompatible again in Version 10.
>
>> Of course, we haven't even started in on the OpenSSL Library fiasco
>> yet (as a concrete example).  The OpenSSL library in the current
>> system doesn't include MD4 - no big deal, until one of the programs in
>> pkgsrc calls for it.  To solve the problem, I install the version from
>> /usr/pkgsrc (which is, IIRC, the same version).  From there, it's just
>> one mysterious library mismatch after another.  My IMP installations
>> fell victim to this one a few weeks ago and I'm still not sure how to
>> fix it.
>
> This is news to me. I thought all that was missing from the base
> system was idea and rc5 support. I'm going to leap to a conclusion
> here: If you're linking against shared openssl libraries in ${PREFIX}
> directly, and also against shared libraries which have embedded ELF
> shared library dependencies on the base openssl libraries, it's
> understandable you'd have problems. Even if I'm wrong in details, the
> mere fact that you're having problems is an argument for fixing the
> base system openssl and marking the package "not for platform
> NetBSD-*-1.6*", not for discarding the base system openssl.
>

One of the choices is to 'discard' the base system executables and
libraries by overwriting them.  Another choise is to rename them.  Either
way, they are out of my tech's way.  This isn't some ethereal, hazy
conceptual argument.  I'm talking about not screwing up people that
actually *USE* the system.
We can discuss the theological implications of the suggestion all we want.
 I pay these guys real money and every time I confuse them by having to
tell them that something that worked yesterday doesn't work today, but
probably will again next week, it costs me money in wasted time.  Every
time I pull up a new library and have to spend hours rebuilding software
because the version of some system library that I need for a new program
is just slight akimbo from one that I was using from the base install. 
There's a thread about the png library running right now.  It's basically
the same thing.
>> Next, we get into Cron jobs - As a default, the crontab path doesn't
>> include /usr/pkg until the end, so any management jobs I start up have
>> to be fully qualified to work.  This works fine until the next
>> upgrade, when I want to switch to the installed routine.  Of course,
>> this really isn't a problem since the installation procedure wipes out
>> the root crontab.
>
> There's already an open PR on that last. [It could be solved by
> letting sysinstall invoke "etcupdate", rather than untar "etc.tgz"
> directly.]
>

I saw that.  There are other ways to solve this one as well - etcupdate on
more than one or two servers gets real old real fast.

> Let's cut to the chase: If the NetBSD userland sources were updated to
> the current level, that would solve all your problems. You could do a
> "cvs update" in "/usr/src", build and install, and then you'd have a
> current system which matched the sources in "/usr/src". That is the
> chief benefit of having a unified system, in contrast with linux,
> where you count on the distributor's package system for all updates.

BINGO!

The only real problems I have with your theory are ones that you provide
below.
>
> The big problem with a policy of relying on pkgsrc for all critical
> updates, is not (simply) that it resembles linux. There are mainly two
> problems:
>
> 1) there's no pressure to update the base system binaries. While it's
> nice not to have pressure, the opensource model relies on users using
> the code and reporting problems and tweaking, and if they don't (or
> can't) use it, it just rots, and
>
> 2) there's no pressure on the pkgsrc people (or the program's authors)
> to fix the binaries to run against the system as released (your MD4
> problem). You start with demanding that openssl be upgraded, and
> before you know it, "this package requires xxx version of libc to
> compile on NetBSD" (shades of linux, again).
>

Please note that you added the concept of "critical updates" to this
debate.  I happen to disagree with your assertion.  Critical updates
should come out as minor releases or "pull-ups" as you call them. 
Optional updates (which probably will but may not be incorporated in the
next release) are the ones I'm talking about.
How bout we add a third possibility to your list?

3) the system assumes that pieces that are in the 'base' installation can
be upgraded through a pkgsrc implementation that recognizes the
distributed system components and allows for a method of importing them.
This abrogates the risk from 1) above because the *point* would be to
manufacture a system that updates the system sources in a repeatable,
predictable way and installs them in the base system.  It also minimizes
the library impact for 2) because (if I remember my shared library theory
correctly) existing exectutables can use a higher minor numbered library
in place of the one that was originally installed.  So, suppose we have a
package that updates some random library.  As long as the update from
pkgsrc doesn't break the minor numbered release, the two libraries can
successfully coexist in the same (/usr/lib) structure.  As far as I can
tell, however, they can't live in different trees in our current ELF
environment.
It also reduces some of the risk from the program side of 2) above.  I can
agree that the packaged binaries could be problematic, but it isn't my
requirement.  I don't have a requirement for binary distributions to work
this way - I recompile from pkgsrc on every server.  Some run 1.5, some
run 1.5.3, since 1.6 Beta, depending on various capabilities and problems
that I've had in the past.  I have a couple of servers whose hardware
didn't reliably in newer versions.  In fact, the reason isn't really
important - the fact that I need a mechanism that's independent of source
version is actually compelling.
By the way, I'm also in the middle of this debate with some folks putting
a DoD system called "DCTS" right now.  Their system is tightly coupled to
specific versions of certain software, and newer versions break the
operating baseline.  The only way to update any software in the system is
to completely reinstall the entire tool suite.  For one server, that's OK.
 For a couple hundred servers, it's nuts!
I still contend that pkgsrc is the right way to do all of this.  Once a
version of the NetBSD source version is released, I want it to stay
immutable and unmodified until the next release (also decidedly not
Linux-like).  The release version should be the most stable, best version
of the software available at the time of release.
If I need something added to my system, I'd like to rely on pkgsrc to get
it there.  Instead of pointing out the option that said I disliked the
most and saying how bad it was, let's stick to one of the ones that I
prefered.
> By the way, there is an option to update "openssh" in the base system
> using pkgsrc -- UPDATE_INTREE_OPENSSH -- but it's a bad hack which sets
> a bad precedent, and it's sure to cause problems down the road. To do
> that with a library like "openssl" is too horrible to contemplate.

I'd agree that this is a bad hack unless it can be extended to handle the
general cases we've discussed above.  As it stands, OpenSSH is just one of
the examples of software we import from other places and include in our
system.  Good examples of the kinds of things that we could stand to have
a mechanism for optional updating include most of the GNU stuff and the
stuff in the dist directory in /usr/src.
As an isolated example, the GNU M4 is required to compile the latest
Sendmail.  The fact that the code finds the right one during installation
is as much a function of the configure script as anything.
Sendmail itself is a good example.  We are using an antiquated version of
Sendmail 8, presumably because the source maintainers have some reason for
not upgrading to the latest version.  The reason isn't important - the
fact that I *CAN* update with the license I purchased means that I should
be able to safely use it.  Soon, Version 10 will be out, and (using the
existing methodology) we will have to wait at least another year before we
will even get Version 9 support.....  Now, I've worked out a way to use
Version 10 (and Version 9 with Milters) that involves deleting the base
programs from the base install.  At best, they're vestigal; at worst, they
can crash my server.
The IPF utility set is another good example of a program that could stand
to be updated more than once a year.
Of course, there are more.  I'm trying to think of a simple,
cost-effective way to update some of these programs without screwing up
the entire source tree and forcing my techs to keep a notebook of where
their programs are today.
I understand most of the reasons why people don't want to do something
like this.  The catch is that I'm old enough now that I couldn't care less
about why things can't be done.   Here's a novel approach - tell me how I
can do what I need to do....  We have at least three different approaches
that work now without forcing me into maintaining a CVS source tree for
every version of the Operating System and every package in pkgsrc
imperpetuity.  Pick one and tell me where the hard parts are.  I'll
stumble around and come up with something that doesn't suck too bad, I
promise.
-- 
Dave Burgess
CTO, Nebraska On-Ramp
Chief Engineer, Mitec Internet Services
Bellevue, NE 68123