[ On Wednesday, May 31, 2000 at 16:13:15 (-0500), Richard Rauch wrote: ]
> Subject: Re: the in's and out's of dependencies between packages...
>
> I've always thought that one of the intended wins of a dynamic library is
> that you can update the library without updating the application.  It's
> kind of sad that we seem to be unable to make them work that way.  To my
> mind, that was generally the biggest value to shared libraries---avoiding
> duplicated code was a comparatively minor feature.

That would be true if only developers were *all* "good enough" to follow
a very strict regime of defining their APIs and always taking great care
to update library version numbers (which of course may not have any
relation to product release numbers); and of course being similarly
careful about using APIs.

I like to compare this problem to the issue of using threads in the
average application.  To really do things properly with threads you have
to think hard about concurrency and that's not something the average
application programmer has been trained to do.

> Do our shared libraries actually (by their construction) prevent such
> transparent upgrades from happening?  Or is it just a result of the way
> that most authors build and maintain shared libraries that you have to
> upgrade everything else whenever you change a shared library?

Well, unfortunately in pkgsrc some libraries are being made into shared
libraries by the pkgsrc maintainers.  This definitely means that unless
the pkgrsrc maintainer becomes very intimate with the code and ensures
that all prototypes are in place, etc., and that all users use the
prototypes properly, etc., and then watches very carefully for any
changes between library releases the only option is to make a new major
release number for every release of the library regardless of whether or
not the API actually changed or not.

Note that this is in effect what happens with some libraries in pkgsrc
now despite the fact that no real attention is paid to versioning the
shared libraries -- the default is to force the user to de-install and
re-install the library when a new version is installed and of course
this often causes every package that depends on the library to also be
upgraded.

Now the proposal I made to allow multiple simultaneous installs of
different versions of every dependant library from pkgsrc on at least
development systems would mostly eliminate the problem.  Any package
could request any version of a library and all would be happy.

Of course this isn't really necessary on production boxes that are only
running "release" bianries and indeed when strictly considering only
shared object libraries it would really only be necessary to ensure that
new major numbers are used for every version of every shared library so
that they could co-exist.  Then if pkg_delete were fixed to never ever
delete any version of a shared library it would be possible to install
new shared object library binaries and new applications that use the new
shared libraries while not having to delete any older library versions
and upgrade the applications that depend upon them.  I think you'd still
need the ability to install multiple simultaneous development libraries
too, else the pkg developer's job turns into a nightmare.

This of course mostly voids most of the reasons one might want to use
shared libraries for, which is why I'd really rather see package
libraries be statically linked with the packages that depend upon them.

Note that none of this necessarily applies for *system* shared object
libraries.  They can still be used and one can still enjoy the ability
to apply binary fixes by simply upgrading a system library and thus
sharing the fix with every binary that uses the shared library.

However that's really not a very effective way to manage binary
installations.  If you look at how it was handled for SunOS-4 you'll
find that what Sun ended up doing was to ship all the ".so" files and
provide programs that allow usrs to replace individual object modules
and rebuild the shared libraries on demand.  They didn't always ship
just the parts and then rebuild the actual libc.so with every patch, but
so far as I recall they did always provide an archive of the individual
modules so that if you'd also applied local patches to other objects
(eg. integrating the resolver into libc) you could still do this and not
clobber the fixes.  What went on behind the scenes is even more
interesting -- Sun were careful to always track every fix by library and
they almost always shipped complete fix sets so that version madness of
libraries didn't ensue.  All of this takes a fairly significant and
central management effort, not to mention custom tools and databases

In the end one really has to wonder whether all this nonsense is really
worth the effort when you've got sources for everything, and especially
when releases are so granular and binary patches are almost non-existant.
Sure you save a bit of disk space with shared libraries, but you rarely
save any CPU cycles (usually you loose lots of them), and there's always
the potential for more security issues to crop up too.

My favourite story about shared libraries and binary installation gone
wrong is from the time I upgraded an AIX-3.1.2 to 3.1.3 or something
similar with only patches (I could get them for free but I couldn't as
easily get the new release at that time).  There were three 2.3GB tapes
(one only half full) in the set, all with a total of at least 20 new
kernels (which required a reboot after each new one) and perhaps as many
as 30-40 new libc's.  The patch tapes were created automatically from a
great big fixes database so that they all applied in the right order and
so that everything was progressively newer (no conflicts) and eventually
what I was left with was indistinguishable (except for the install logs)
from the new release (or so I was told).  It was a fascinating, but
eventually boring, process, and not something I *ever* want to have do
again!  ;-)

Who needs all the big databases and tracking systems and all the messy
details that have to be handled by hand?  It's an accounting nightmare!
At least if you build from source and install source patches it's
relatively hard to get unsolvable conflicts between fixes and not too
hard to cut new releases and do mostly automatic "micro-upgrades" if
you've got a whole farm of similar machines.


-- 
							Greg A. Woods

+1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>