[ On Wednesday, September 20, 2000 at 04:06:11 (-0700), John Nemeth wrote: ]
> Subject: Re: /etc/default ickiness...
>
>      I don't believe I am doing that.  You keep trying to convince me
> to use your methods.  I've used all sorts of methods and I know what
> works for me and what makes life harder for me.  We'll probaby never
> agree on methods, so why don't we just agree to disagree and you can
> stop trying to convince me to use your methods, which is an exercise in
> futility?

I've been there, and done that for over a dozen years now.  I *know*
you're making your life harder because I've been through exactly the
same things and made my one life harder in exactly the same ways!  :-)

Now I'm working to try and make my life, and your life, easier!
Obviously I can't force you do to anything in any particular way but
hopefully in this dicussion I can at least provide a few ideas to other
people and perhaps even help myself work out some of the issues more
clearly in my own mind.

>      With patch releases, if I do them at all, I usually don't bother
> with the /etc set; since patch releases aren't supposed to change
> functionality there is usually nothing of interest changed in the /etc
> set.

Don't bet on it....  Any given system might not depend on those changes
at the time the upgrade is performed, but systems are dynamic living
things and who's to say that a week later you might need exactly the fix
that was in some /etc file from the upgrade?

> } point releases there are likely to be so many new things that you'll at
> } least want to check the new defaults and no doubt some variable names
> } may even have been deprecated too, etc.)
> 
>      Correct.  So much for not having to do anything...

It's one thing to read through the various critical files, diff them,
etc., and quite another to have to re-configure everything from scratch!

>      This sounds like a lot of work for no particular reason.

The effort is necessary in order to get to the first step in the process
of making things easier down the road.  This little bit of effort up
front is what saves enormous complications and contortions later on.  If
you don't believe in preparing ahead of time to make your life easier
down the road then obviously this won't make sense, but if I can forsee
something being difficult down the road then I'm going to do what I can
to prepare and make things as easy as possible down the road.  It's
amazing what one can learn from the school of hard knocks if one pays
even the slightest amount of attention in class (i.e. "Real Life(tm)")!

Note that the very minimal effort of doing as I say is only made
"difficult" by trying to do it using difficult-to-use antiquated
editors.  Any modern multi-window text editor solves those problems so
completely that there is absolutely zero additional effort required!
I'm told that even "vi" can copy lines from one file to another with
relative ease (you just can't see both files at the same time if you've
only one terminal session).

BTW, once you've done it a couple of times then you'll really appreciate
just how easy it is to do too.

>  Given
> that I will always need to at least examine the /etc/default files, I
> may as well have all the info in a convenient place (i.e. in the /etc
> files).

The trick is in reducing the amount of information that you have to
manage from being *everything* to being just those items that make your
system unique from the distribution.

>  That way, it will be fairly easy to develop a tool to do a
> three way merge between the old /etc/orig file, the new /etc/orig file,
> and my changed /etc files.  After the automated merge is done, I will
> simply have to quickly check the resulting new /etc files for any new
> variables that I may want to changed.  This could also be helped by
> flagging the new variables (the three way merge tool would preprocess
> the /etc files to remove the NEW flag before using it).

One of the reasons I've not jumped immediately into the fray and written
a simple tool to keep track of ancestor versions and do such merges is
that the tools we have to do these kinds of things (i.e. diff3) are not
perfect.  As anyone who's done much with CVS and branch merges can
attest to, automated merges can just as easily lead to more effort being
required on behalf of a human as they can to eliminate the need for
human intervention.

Meanwhile the trick employed by NetBSD with rc.conf (or in the past by
rc.conf.local) is immediately and instantly usable by any human to
understand what's been done to configure a particular system.

> } There's literally no real difference between your /etc/orig and the
> } existing /etc/default (except that one could in theory make changes to
> 
>      Actually, there is a real difference.

What I meant was that fundamentally the result is exactly the same.  Two
systems configured with each method will behave identically.

The differences are not operational but rather in the techniques used to
manage them.

>  I don't need to go through
> extra unnecessary steps to see the stuff that I want to change, or to
> find out the value of a variable at a later time (with /etc/default, I
> would first have to look in the /etc file to see if I gave it a
> non-default value, and if not, then look in the /etc/default file).
> One thing that people being taught about databases are taught early on,
> is that having the same information in multiple places without an
> automated replication scheme is a recipe for disaster.  rc.conf could
> conceptually be thought of as database that describes system startup.
> You now have to look in two places to figure out how the system startup
> is configured, instead of just one.  This is a very bad thing.

Your /etc/orig scheme, or use of a version control tool (eg. RCS), is
also replicating the same information.  The only difference is that such
schemes don't make live use of the replicated information.  In fact the
replication of the configuration item sets in their entirety is what's
necessary to be able to re-discover the uniqueness of a given version
after the fact.

In fact automating the replication is what makes it *safer*!  The use of
RCS is by and far the best way to replicate this information of course
since it makes it impossible to accidentally change the original version
thus ensuring that the correct delta can always be calculated at any
time in the future.

With the /etc/default scheme the /etc/rc.conf file is the delta.  With
RCS you've (conceptually) got the delta and the original stored in the
,v file.  With your /etc/orig scheme the delta must be re-generated by
comparing the original with the modified copy.

The advantage of the /etc/default scheme is that you can swap original
/etc/default files around underneath the delta file (under controlled
upgrade procedures of course) and not necessarily have to do anything
else to retain the uniqueness of the system instance.

Certainly there are changes that can be introduced by the change of
system defaults, but those changes won't always have a negative effect
on the way a given instance of a system fulfills its requirements.  Say
for example that SSH becomes a built-in tool and perhaps the system
didn't have an add-on SSH on it already, so in that case the
introduction of a new daemon is probably a very positive and beneficial
change in the default behaviour of the system.

> } The /etc/default scheme does not make creation of a merge tool any
> } easier at all in fact.  What it does is to make upgrades much easier
> } *without* a merge tool.
> 
>      Not really, because I would still have to do a detailed comparison
> between the old /etc/default files and the new /etc/default files, then
> possibly make changes to the /etc files.  When doing that, it makes no
> difference where the information is, except that if you increase the
> number of files that I must process then I'm going to be very annoyed
> (this is one of the things I really hate about SysV based systems).

It all depends on how well you already understand the system and how
much you trust the vendor.  Yes you might want to diff /etc/default/*
with /etc.old/default/* just to get the warm fuzzies, but this is so
trivial to do it's just not funny.  You don't have to do any edits if
you don't need to.

In any case simple deductive logic proves my point.  Without an
/etc/default style of system it's impossible to just upgrade and go
because the upgraded /etc/rc.conf file must always be re-edited to
re-introduce local configuration changes.  Therefor one must conclude
that with at least the possibility of being able to upgrade and go
without having to reconfigure anything must in fact make the
/etc/default scheme easier to use *without* a merge tool.

What's interesting here though is that although rc.conf is probably one
of the few types of configuration information which can be managed in
this way without needing to use an /etc/orig or RCS style of tracking,
it's also critical and common enough information that having a short,
concise, list of just the unique items is useful in and of itself.  I
use RCS on /etc/rc.conf (and did on rc.conf.local too).

None of this stuff matters a whole lot if you've got only one system to
manage through a series of upgrades.  However if you've got dozens of
such systems, each with unique configurations, this particular 

Yes, a merge tool is eventually necessary for all the other kinds of
configuration files in /etc anyway, so we still need one in the long
term, but I won't give up on /etc/default once I have a merge tool
because of all these other benefits it has.

> } In the specific case of a localised /etc/rc.conf that only modifies the
> } values inherited from a master /etc/default/rc.conf there is perhaps
> } some need for a tool that can verify that an upgrade does not change a
> } configuration.  Such a tool is almost too trivial to write though since
> } any experienced person should be able to invent it on the spot, but
> } since it needs a few temporary files it might best be turned into a
> } script:
> } 
> } 	: ${TMPDIR:=/var/tmp}
> } 	( set > $TMPDIR/$$.base )
> } 	( . /etc/rc.conf ; set | cat - $TMPDIR/$$.base | sort | uniq -u > $TMPDIR/$$.oldconf )
> } 	mv /etc/default/rc.conf /etc/default/rc.conf-OLD
> } 	mv /etc/default/rc.conf-NEW /etc/default/rc.conf
> } 	( . /etc/rc.conf ; set | cat - $TMPDIR/$$.base | sort | uniq -u > $TMPDIR/$$.newconf )
> } 	( . /etc/default/rc.conf-OLD ; set | cat - $TMPDIR/$$.base | sort | uniq -u > $TMPDIR/$$.olddef )
> } 	( . /etc/default/rc.conf ; set | cat - $TMPDIR/$$.base | sort |	uniq -u  > $TMPDIR/$$.newdef )
> } 	comm -13 $TMPDIR/$$.oldconf $TMPDIR/$$.newconf > $TMPDIR/$$.chgvar
> } 	comm -13 $TMPDIR/$$.olddef $TMPDIR/$$.newdef > $TMPDIR/$$.newvar
> } 	comm -23 $TMPDIR/$$.chgvar $TMPDIR/$$.newvar
> 
>      $$.oldconf and $$.newconf are exactly the same, so the first comm
> is meaningless.

Ah, no, $$.oldconf and $$.newconf are not necessarily the same, and
that's the key to understanding what's going on!  You say as much
yourself below so perhaps you do understand but my attempt to express my
idea in a computer readable manner didn't elucidate it well enough for
you.

What I'm trying to find is the set of items which are changed in the
overall configuration, and the set of items which are changed in the
default configuration and then find the set of items which are only part
of the first set, i.e. those configuration items which changed.
Obviously the second set is of some interest too since it identifies the
items which are different in in the default configuration but if we
assume that the new defaults are generally "good for all" then I don't
think they're quite as important.

>  Although, I do see the gist of what you are trying to
> accomplish.
> 
> } If I'm not mistaken (the above is not tested in any way and my set
> } theory is a bit rusty it seems), the output should be just the default
> 
>      sh script programming has never been one of my strong points.  All
> the common scripting languages are horribly convoluted (especially
> PERL).  Anybody interested in standardising on REXX, or at least making
> it a common scripting language (how can you tell that I worked on IBM
> mainframes before doing UNIX)?

Yes, Perl is especially convoluted, but....

The above isn't really "sh" programming -- the only shell trick is in
using the shell and "uniq" to display just those the variable settings
that are changed by a script.  It is of course Unix programming in the
true sense of using basic filters and tools though.

Any language that can easily manipulate lists of things and do set
operations on them would achieve the same result once given the lists of
"variable=value" pairs.  I'd have written my example in lisp if I were
just wanting to get the job done since that's the language I first
learned to apply set theory with, but since there's perhaps some desire
to do this with just the tools that would be available on a plain unix
system, and given the audience in this forum, I chose to use them and
they are more than adequate for the task.

> } configuration items changed by the upgrade and not already overridden by
> } local settings, excluding any new items that would not have been
> } mentioned in the old files.  In an upgrade the output should probably be
> } empty in fact.  If you want to change the defaults in any of the new
> 
>      You may think that; however, default variables are changed between
> point releases for various reasons (i.e. tightening up security,
> enabling things that were previously experimental, etc.).

Yes, which is exactly why I compare the new and old default
configurations in the sample above....

-- 
							Greg A. Woods

+1 416 218-0098      VE3TCP      <gwoods@acm.org>      <robohack!woods>
Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>