Subject: Google SoC 2007 Project: Subsystem Conditional Building (resend)
To: None <tech-userlevel@netbsd.org>
From: Brian A. Seklecki <lavalamp@spiritual-machines.org>
List: tech-userlevel
Date: 03/19/2007 20:47:08
[some format/typos corrections from pasting into MUA corrected]

Note:  This is strictly an informal discussion; not a full proposal at this 
point.

Some of you may know that I'm starting a small project called 'bsd-appliance' 
to hopefully help centralize and coordinated embedded technology related 
development for the *BSD families (a la linuxdevices.com).  I've released a 
small set of tools (framework/toolkit) to help build and assemble dynamic file 
systems and bootable file-system images following a strategy we developed at 
Collaborative Fusion, Inc.

As an extension of that project, I'd like to propose this SoC project as 
(hopefully) a catalyst for additional projects.

Background:

One of the topics that the framework does not address is _what_ file system 
objects to include in such images (we just say 'pull from this $DESTDIR').  As 
you know, many resource-constricted embedded platforms rely on translating 
archived read-only userland data on CF media into memory-backed filesystems 
using mfs(4) and md(4)/rd(4).

Profiling the system is really beyond the mandate of that framework.  It is 
also very implementation-specific, so the project only offers advice on how to 
do so.  However, every time I give someone the advice: "Just extract base.tgz 
and etc.tgz and create a list of binaries and libraries relative to / and /usr 
in rd_prune.txt and cf_prune.txt respectively to prune automatically", I cringe 
because I know that this is the least graceful method of accomplishing this 
goal.

Proposal:

I propose, a interim solution to syspkg, making a number of additional 
subsystems conditionally built using the mk.conf(5) variable method. This 
approach, although obviously much less clean and efficient than a the syspkg 
approach, is beneficial in many ways none the less.  For starters, it begins to 
identify subsystems, their respective $DESTDIR and source tree objects, and 
their sizes.

Conversely, the lack of scalability of this approach in the long-term becomes 
apparent based on the need, for each system conditionalized, to touch so many 
other systems related to the build.

For each subsystem, I will follow the examples in the already conditionalized 
subsystems (list below).  The changes frequently reach into:

- Makefile $SUBDIR conditions
- etc/${subsys_name} Entries
- etc/mtree/set.* Entries
- etc/rc.d/ Entries
- postinstall(8) checks
- usr/share/examples/* contents
- usr/share/doc/* contents
- usr/share/man/whatis*db Entries
- Further categorizing in distrib/sets/lists/*/mi
- Further categorizing in distrib/syspkg/sets/

Already conditional: GNU CVS, GCC, GDB, ipf(4) / ipfilter, kerberos, lint, pam, 
OpenBSD pf(4), Postfix, S/Key, YP, Hesoid.

These are a great start.  For this project, I've earmarked the following 
additional subsystems:

tcpdump(8) / libpcap, GNU diff(1)/patch(1), ISC NTP, ISC DHCP, RAIDFrame 
(userland), LFS (userland), systrace (userland), ISDN4BSD, CGD (userland), CCD 
(userland), AltQ (userland), WPA Supplicant (userland), HostAP, Racoon, PPP / 
SLIP / PPPOE, LP Utilies, SUP, Kerberos (userland), NFS (userland), Quota 
Utilities (userland), AMD, OpenSSH, iSCSI/libiscsi (userland), libradius (pam), 
libevent, libprop

Below is a cursory summary of the most of these.  Of course, the reality 
if conditionalizing these is likely to be a series of interdependencies 
that only the userland maintainer knows of.

The sum of the space savings of the below systems (by no means comprehensive) 
approaches 8mb (~7382748 K).  At first glance, this project proposal may seem 
like an awful lot of work to save 8mb in $DESTDIR, however when working on 
embedded hardware platforms with often cumbersome memory limitations (32, 64, 
128), 8mb can make a world of difference.

Additionally hopefully we can create a precedent for modularizing additional 
features that require new userland utilities and libraries.

~BAS

---------------------------------------------------------
Subsystems Overview:
---------------------------------------------------------


Subsystem Name: tcpdump(8)/pcap(3)/libpcap
Userland: Yes
Kernel: Yes (not affected)
Source tree objects:
  usr.sbin/tcpdump/
  dist/tcpdump
  dist/lib/libpcap
File system objects:
  /usr/lib/libpcap.a
  /usr/lib/libpcap.so
  /usr/lib/libpcap.so.2
  /usr/lib/libpcap.so.2.0
  /usr/lib/libpcap_p.a
  /usr/lib/libpcap_pic.a
  /usr/sbin/tcpdump
Estimated size K (i386): 1195235

Subsystem Name: raid(4)
Userland: Yes
Kernel: Yes (Not related)
Source tree objects:
  sbin/raidctl/
File system objects:
  /sbin/raidctl
Estimated size K (i386): 24892

Subsystem Name: isdn(4)
Userland: Yes
Kernel: Yes (no related)
Source tree objects:
  usr.sbin/isdn
  share/examples/isdn/
  share/man/man4/isdn*
File system objects:
  /usr/sbin/isdnd
  /usr/sbin/isdnmonitor
  /usr/sbin/isdntel
  /usr/sbin/isdntelctl
  /usr/sbin/isdntrace
Estimated Size in K (i386): 295032

Subsystem Name: LFS
Userland: Yes
Kernel: Yes (not affected)
Source tree objects:
  usr.sbin/dumplfs
  sbin/mount_lfs
  sbin/newfs_lfs
  sbin/resize_lfs
  libexec/lfs_cleanerd
File system objects:
  /libexec/lfs_cleanerd
  /sbin/mount_lfs
  /sbin/newfs_lfs
  /sbin/resize_lfs
  /usr/sbin/dumplfs
Estimated Size in K (i386): 148934

Subsystem Name: systrace(4)
Userland: Yes
Kernel: Yes (Not Affected)
Source tree objects:
  bin/systrace
File system objects:
  /bin/systrace
Estimated Size in K (i386): 151671

Subsystem Name: cgd(4)
Userland: Yes
Kernel: Yes (Not Affected)
Source tree objects:
  sbin/cgdconfig
  share/man/man4/cgd.4
File system objects:
  /sbin/cgdconfig
Estimated Size in K (i386): 41490

Subsystem Name: ccd(4)
Userland: Yes
Kernel: Yes (Not Affected)
Source tree objects:
  sbin/ccdconfig
  share/man/man4/ccd.4
File system objects:
  /sbin/ccdconfig
Estimated Size in K (i386): 15513

Subsystem Name: altq(9)
Userland: Yes
Kernel: Yes (Not Affected)
Source tree objects:
  usr.sbin/atlq
File system objects:
  /usr/sbin/altqd
  /usr/sbin/altqstat
Estimated Size in K (i386): 114200

Subsystem Name: ISC DHCP
Userland: Yes
Kernel: No
Source tree objects:
  dist/dhcp
  usr.sbin/dhcp
File system objects:
  /sbin/dhclient
  /sbin/dhclient-script
  /usr/sbin/dhcpd
  /usr/sbin/dhcrelay
  /usr/share/man/cat5/dhcp-eval.0
  /usr/share/man/cat5/dhcp-options.0
  /usr/share/man/cat5/dhcpd.conf.0
  /usr/share/man/cat5/dhcpd.leases.0
  /usr/share/man/cat8/dhcpd.0
  /usr/share/man/man5/dhcp-eval.5
  /usr/share/man/man5/dhcp-options.5
  /usr/share/man/man5/dhcpd.conf.5
  /usr/share/man/man5/dhcpd.leases.5
  /usr/share/man/man8/dhcpd.8
Estimated Size in K (i386): 1431563

Subsystem Name: ISC NTP
Userland: Yes
Kernel: Yes (Not Affected)
Source tree objects:
  dist/ntp
File system objects:
  /usr/sbin/ntp-keygen
  /usr/sbin/ntpd
  /usr/sbin/ntpdate
  /usr/sbin/ntpdc
  /usr/sbin/ntpq
  /usr/sbin/ntptime
  /usr/sbin/ntptrace
  /usr/sbin/sntp
Estimated Size in K (i386): 777795

Subsystem Name: wpa_supplicant(8)
Userland: Yes
Kernel: No
Source tree objects:
  dist/wpa_supplicant
File system objects:
  /usr/sbin/wpa_cli
  /usr/sbin/wpa_passphrase
  /usr/sbin/wpa_supplicant
Estimated Size in K (i386): 715922

Subsystem Name: hostapd(8)
Userland: Yes
Kernel: No
Source tree objects:
  usr.sbin/wpa/hostapd
  usr.sbin/wpa/hostapd_cli
  share/examples/hostapd
File system objects:
  /usr/sbin/hostapd
  /usr/sbin/hostapd_cli
Estimated Size in K (i386): 189744


Subsystem Name: ipsec-tools / racoon(8)
Userland: Yes
Kernel: Yes (Not Affected)
Source tree objects:
  crypto/dist/ipsec-tools
  share/examples/racoon
  usr.sbin/racoon
  usr.sbin/racoonctl
File system objects:
  /usr/sbin/racoon
  /usr/sbin/racoonctl
Estimated Size in K (i386): 563791

Subsystem Name: sl(4)/ppp(4)/pppoe(4)
Userland: Yes
Kernel: Yes (Not Affected)
Source tree objects:
  sbin/slattach
  sbin/pppoectl
  usr.sbin/sliplogin
  usr.sbin/pppd
  dist/pppd
  share/man/man4/*ppp*
File system objects:
  /sbin/pppoectl
  /sbin/slattach
  /usr/sbin/pppd
  /usr/sbin/pppdump
  /usr/sbin/pppstats
  /usr/sbin/sliplogin
  /usr/sbin/slstats
Estimated Size in K (i386): 346654

Subsystem Name: lp(4) and friends
Userland: Yes
Kernel: No
Source tree objects:
  usr.sbin/lpr
  usr.sbin/lpctl
File system objects:
  /usr/bin/lp
  /usr/bin/lpq
  /usr/bin/lpr
  /usr/bin/lprm
  /usr/bin/pr
  /usr/sbin/lpc
  /usr/sbin/lpd
  /usr/sbin/lptest
Estimated Size in K (i386): 216203

Subsystem Name: sup(8) / supservers(8)
Userland: Yes
Kernel: No
Source tree objects:
  usr.sbin/sup/
  share/examples/supfiles
File system objects:
  /usr/sbin/sup
  /usr/sbin/supfilesrv
  /usr/sbin/supscan
  /usr/share/examples/supfiles/sup.jp.netbsd.org
  /usr/share/examples/supfiles/sup.netbsd.org
  /usr/share/examples/supfiles/sup2.fr.netbsd.org
Estimated Size in K (i386): 151497

Subsystem Name: quota(8)
Userland: Yes
Kernel: No
Source tree objects:
  usr.sbin/quotaon
  usr.sbin/repquota
  usr.sbin/quotacheck
  usr.sbin/edquota
  usr.bin/quota
  libexec/rpc.rquotad/
File system objects:
  /usr/libexec/rpc.rquotad
  /usr/sbin/edquota
  /usr/sbin/quotacheck
  /usr/sbin/quotaoff
  /usr/sbin/quotaon
  /usr/sbin/repquota
Estimated Size in K (i386): 90902

Subsystem Name: amd(8)
Userland: Yes
Kernel: No
Source tree objects:
  usr.sbin/amd
  share/examples/amd
File system objects:
  /usr/sbin/amd
Estimated Size in K (i386): 146519

Subsystem Name: amd(8)
Userland: Yes
Kernel: No
Source tree objects:
  usr.sbin/amd
  share/examples/amd
File system objects:
  /usr/sbin/amd
Estimated Size in K (i386): 146519

Subsystem Name: iscsi-target(8) / libiscsi
Userland: Yes
Kernel: Yes (Not affected)
Source tree objects:
  dist/iscsi/
  lib/libiscsi
  usr.sbin/iscsi
File system objects:
  /usr/lib/libiscsi.a
  /usr/lib/libiscsi.so
  /usr/lib/libiscsi.so.0
  /usr/lib/libiscsi.so.0.0
  /usr/lib/libiscsi_p.a
  /usr/lib/libiscsi_pic.a
  /usr/sbin/iscsi-target
Estimated Size in K (i386): 618672

7~

l8*
 	-lava (Brian A. Seklecki - Pittsburgh, PA, USA)
 	       http://www.spiritual-machines.org/