Re: RFC: boot images config. file format

[Masao Uebayashi <uebayasi%gmail.com@localhost>]

Hi,

Thanks for bringing this up.  I have (too) many comments ... can't
write all right now.

a) Be aware of sysinst

As you wrote some time ago, off-line installation and on-line
installation (sysinst) should be considered together.  Share
infrastructure as much as possible.  What are the internal interfaces?

b) Filesystem content vs. disks/partitions

I don't like these two things are mixed in single configuration file.
Please separate them clearly.

c) Multiple disks, raid, LVM, ...

We should be able to handle multiple disks.  For example 1GB USB disk
as system (/) and multiple HDDs as data.  A small NOR FlashROM as /
and a larger NAND FlashROM as /usr, etc.

Hopefully the config file can be capable of raidframe and LVM too.

d) Configuration vs. customization

I want to clearly separate configuration and "customization" (my
terminology; these are not clearly defined in NetBSD!)

Editing /etc/rc.conf and removing part of /bin/* (base-util-bin) are
very different; the former is a standard configuration.  The latter is
beyond the standard configuration; we don't expect *usual* users to do
that.  Thus I'd call it "customization", meaning user makes an
unexpected local modification.

If local changes are only to add files in some unused filesystem
namespace (typically /usr/local or /home/*, etc), we could think of
these changes as not really unexpected.  These changes are less likely
to harm the default behavior.  I think we should define the standard
interface for users to extend freely (which is what I intended by
"extsrc").

I think the customization part of the proposed statements (exclude,
files, patches, trim, build, ...) is too flexible.  Again, I want to
clearly differenciate these local changes.  If an image has nothing
locally changed, but only install some syspkgs, it's a perfectly
normal NetBSD system.

Masao

On Mon, Apr 12, 2010 at 4:19 AM, David Young <dyoung%pobox.com@localhost> wrote:
> Please let me know what you think about this configuration file syntax
> for NetBSD boot images.  The format derives from Lloyd Parkes' and my
> work together on GSoC 2009.
>
> ENTITIES
>
> The entities specified by the configuration file specifies, in order
> from the "atoms" to the final product, a disk image:
>
> <fstab>: a list of filesystems to list in /etc/fstab.  An fstab is
>    created with an 'fstab' statement.  An fstab is instantiated at
>    /etc/fstab by including it in a <list>.
>
> <passwd>: a list of password entries to add to the system.  A <passwd>
>    is instantiated in files (/etc/passwd, /etc/master.passwd,
>    /etc/pwd.db, /etc/spwd.db) by including it in a <list>.
>
> <group>: a list of group entries to add to the system.  A <group>
>    is instantiated in files (/etc/group) by including it in a <list>.
>
> <list>: a list of directories, files, and the patches to apply to the
>    files.  A list is built by adding/excluding syspkgs, adding files,
>    copying another list, by installing /etc/fstab or /netbsd, and
>    by specifying applicable patch directories.  Statements that can
>    operate on or create a list include:
>
>        syspkgs: add one or more syspkgs to a list
>        exclude: delete one or more syspkgs from a list.
>        copy: copy a list, possibly adding /netbsd, /etc/fstab,
>            /etc/group, or password files
>        files: add files listed by an mtree(5)-format file to a list
>
>    See STATEMENTS.
>
> <filesystem>: a subset of a <list>, rendered into a particular
>    filesystem format.  The files and directories in a <list> are
>    divided among its filesystems so that the files and directories
>    in 'filesystem /' will not appear in 'filesystem /var'.
>
> <partition>: a copy of binary data such as a kernel, filesystem, or
>    ordinary file, with disk parameters such as size, offset, and label
>    specified.
>
> <disk>: an instantiation of zero or more <partitions> and disk partition
>    tables in a disk image with a certain size and C/H/S geometry.
>
> <tarroot>: an instantiation of a list in a .tgz file
>
> <mdroot>: a copy of a kernel with a particular filesystem loaded
>    into an embedded memory disk.
>
> Configuration entities build up to a disk image roughly as follows:
>
> file -> fstab | kernel | plainfile | group | passwd
> list -> file | file list
> filesystem -> list path fstype
> partition -> filesystem | file
> disk -> partition* bootopts
>
> STATEMENTS
>
> Here is the syntax for configuration file statements:
>
> #
> # filesystem <fsname> [at <path>] list <listname> type ffs|cd9660 \
> #     [size <blocks>] <TBD: specification of makefs options>
> #
> Create a filesystem named <fsname> with the given type and (optional)
> size, and fill it with files from <listname>.  If 'at <path>' is
> given, fill the filesystem only with files in <listname> under
> <path>. <path> must be a pathname starting with '/'. If <fsname>
> is a pathname starting with '/', fill the filesystem only with
> files in <listname> under path <fsname>.  If neither <path> nor
> <fsname> is a pathname starting with '/', the filesystem statement
> is invalid.
>
> #
> # partition <partname> [start <block>] [size <blocks>] \
> #   [bsd [part <letter>] fstype <type>] [gpt type <UUID> [name <name>]] \
> #   [mbr [part [ext] 0|1|2|3] [sysid <id>] [active]] \
> #   { kernel <kernelname> | filesystem <fsname> | file <filename> | zero }
> #
> Create a partition named <partname>.  Optionally set its parameters in
> the BSD disklabel, Master Boot Record, and GUID Partition Table.  Fill
> it either with zeroes or with the given kernel, filesystem, or ordinary
> file.
>
> #
> # disk <diskname> sectors <sectors> \
> #    [geometry <cylinders> <heads> <sectors/track>] {partition <partname>}*
> #
> Write a disk image to <diskname>.img. <diskname> is not a disktab(5)
> entry name.
>
> #
> # tarroot <tgzname> list <listname>
> #
> Create a gzipped tape archive file, <tgzname>.tgz, and add to it
> all of the files in <listname>.
>
> #
> # mdroot <mdname> kernel <kernelname> filesystem <fsname>
> #
> Copy the kernel configuration <kernelname>.  Configure the copy to
> mount the root filesystem from an embedded memory disk.  Build a
> kernel, <mdname>, from the copied configuration, and embed the
> filesystem <fsname>.
>
> #
> # fstab <fstabname> {filesystem <fsname> | nfs <server:/path>} \
> #     at <mountpoint> [<fstab fields 4 through last>]
> #
> Create or extend a list of fstab(5) entries, <fstabname>.  <fstabname>
> will not appear on any filesystem unless it is added to a files
> list with a 'copy' statement (below), and the files in the list
> are added to a filesystem with a 'filesystem' statement.
>
> #
> # copy <new-listname> list <listname> [fstab <fstabname>] [netbsd 
> <kernelname>]
> #
> Create a copy of a files list, <listname>.  Call it <new-listname>.
> Either compile <kernelname> and it add to the list at /netbsd, add
> <fstabname> to the list at /etc/fstab, both, or neither.
>
> #
> # syspkgs <listname> <syspkg-regex>
> #
> Add to <listname> all of the files from System Packages matching
> the regular expression <syspkg-regex>.
>
> #
> # exclude <listname> <syspkg-regex>
> #
> Remove from <listname> all of the files from System Packages matching
> the regular expression <syspkg-regex>.
>
> #
> # files <listname> <directory>
> #
> Add to <listname> all of the files listed in the mtree(5)-format
> file, <directory>/METALOG.  Use the file ownership, mode, and type
> specified by METALOG.  Take each regular file's content from the
> corresponding file under <directory>.
>
> #
> # patches <listname> <directory>
> #
> Treat each file under <directory> as a patch.  Apply the
> patch at the root of the list <listname>.
>
> #
> # trim <listname> <trimfile>
> #
> Read from <trimfile> a newline-delimited list of files to remove
> from <listname>, and remove them.
>
> #
> # build <listname> <srcpath>
> #
> Cross-build the 3rd-party software at <srcpath> by changing to the
> directory <srcpath> and invoking the targets obj, dependall, and
> install of the top-level Makefile.  Add the installed files to
> <listname>.
>
> #
> # user <passwdname> <record>
> # group <groupname> <record>
> #
> Add the passwd(5)/group(5)-format <record> to the list of password/group
> records named by <passwdname>/<groupname>.
>
> I have attached a sample configuration file that is adapted from
> configuration file written by Lloyd Parkes for a Kerberos Key
> Distribution Center (KDC).
>
> Dave
>
> --
> David Young             OJC Technologies
> dyoung%ojctech.com@localhost      Urbana, IL * (217) 278-3933
>