Re: ports

[Izaac <izaac%setec.org@localhost>]

On Fri, Aug 31, 2012 at 03:30:29PM -0500, Jeremy C. Reed wrote:
> On Fri, 31 Aug 2012, Izaac wrote:
> 
> > All that aside: I'm no apologist for pkgsrc.  I frankly think it's
> > pretty awful and not a little embarrassing.  But this isn't one of the
> > things it gets entirely wrong.
> 
> I have been studying variety package systems for over a decade on 
> different systems and consistently keep finding that overall pkgsrc has 
> been the best for my needs. Can you please share what is awful, 
> embarrassing, or wrong?  Let's improve it.

"No one pretends that democracy is perfect or all-wise. Indeed, it has
been said that democracy is the worst form of government except all
those other forms that have been tried from time to time."
                                    - Sir Winston Churchill, 1947-11-11

Wise words.  And pkgsrc is very similar to democracy in this case.  Of
those systems with which I have had experience, it is the most
versatile, intuitive, and frankly pleasing.

However.

You ask a perfectly valid question, and it deserves a real response.  I
unfortunately don't have the time right now to perform the appropriate
analysis.  So here's a few things off the top of my head:

First, let us consider the raw stuff of pkgsrc, i.e. the hierarchy and
files themselves.  At present, pkgsrc-current consists of 119436 files
filling 703MiB of disk.  Why so enormous?  Because 84875 of these files
are one 512-byte block or less in length.

Is CVS to blame?  Partially, sure.  But let's remove it.  The statistics
are now: 70112 files occupying 447MiB where 36132 are one 512-byte block
or less.  (But now we can't update.  Not bad for quarterly release,
except that we release CVS in the tarball anyway.)  This is pretty
insane, considering that the contents of these files -- sans CVS --
amount to 159MiB of actual data; a 2.8x inflation.

[Yes, I recognize the advantages of a consistent, easy-to-manipulate,
easy-to-version-control, plain layout.  I enjoy it.  The price is easy
to pay.]

Second, make(1) is a poor choice for resolving dependencies.  In
considering that we have all these various files laying about, resolving
dependencies is a non-trivial task.  Allow me to present an example by
performing a build of what ought to be a very simple, dependency-free
package, i.e. misc/figlet.  The list of Makefiles to check, i.e.
.MAKE.MAKEFILES, winds up being 21.  Twenty-one files are read, parsed,
and the various variables are checked, set, unset, modified, and
whatever else necessary to even approach the actual building of a really
simple package.  All of this setup is not free by any stretch of the
imagination.  It becomes blatantly obvious on even the swiftest machine
when attempts to do something as simple as 'make clean' in, say,
www/firefox:

   # ( cd /usr/pkgsrc/www/firefox && time make clean )
   102.224u 44.366s 1:29.92 163.0% 0+0k 305+0io 15pf+0w

Madness.  It's why pkgtools/pkgclean even exists.

[Yes, I recognize that using a single tool which maintains a single
state for the entire process from configuration, through patching,
building, installation, and final packaging is remarkably powerful.  But
can't we maybe cache some of the actual dependency graphing for at least
the quarterlies?  A hints.mk which can be pulled and tucked into
/var/db/pkg somewhere?  I had a little luck doing something like this
for one-offs.]

Third, binary package handling.  Have you ever actually tried something
like:

   # export PKG_PATH 
http://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/5.1_2012Q2/All/
   # pkg_add -v calibre

The verbose does absolutely nothing to convey the fact that something is
being downloaded (presumably in order to read its +FILEs?,) which is why
the program is just sitting there silently.  And then it love to
repeatedly advise that:

   pkg_add: Warning: package `thinyoucareabout-3.14.159nb2718' was built for a 
platform:
   pkg_add: NetBSD/x86_64 5.1 (pkg) vs. NetBSD/x86_64 5.1.2 (this host)

(Why, pray tell, don't we have binary packages that are actually built
for the latest release version of the 5.1 branch, i.e. 5.1.2?  4.0.1 is
correct.  5.0.2 is correct.  Hell, there's even a 6.0_RC1.  And don't
get me started on the string inanities of "amd64" and "x86_64".)

(Also, build reproducibility.  There's been some interest in
reproducible binaries in NetBSD in general.  In pkgsrc, it's
particularly infuriating to find that the only thing preventing this for
many packages is BUILD_DATE in +BUILD_INFO.  The ability to retrieve and
compare +BUILD_INFO and +CONTENTS would be immensely useful to that end.
Imagine if it could be used to resolve some of the make package
conflicts as well!)

And woe unto ye if you want to try and maintain bulk binary updates.
The only real tool we have for this kind of thing is pkg_chk.  And it's
not pkg_install.conf friendly.  Nor does it understand PKG_PATH, if
memory serves.  I hate to even offer it as a comparison, but yum
::turns, spits:: really does have that whole enterprise-class repository
thing pretty much down (obvious, SERIOUS caveats aside.)

[Yes, I agree that updates are so critical to security that responsible
administrators should be doing these things via manual selection and
pkg_add -u.  And yet.]

Fourth, database integrity.  If I had a nickel for every time I've had
to pkg_admin rebuild / pkg_admin rebuild-tree because something died
along the way.

And more, but they devolve to nits.  Again, this question is deserving
of a more serious amount attention.  And I've been trying to make
efforts to address many of these issues.  But time is a scarce commodity
for me right now.  For that, I apologize.  Perhaps, though, this is a
start for consideration.

-- 
. ___ ___  .   .  ___
.  \    /  |\  |\ \
.  _\_ /__ |-\ |-\ \__