Re: RFC: checkflist extra prefixes and post-build hooks

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Fri, 18 Apr 2025 17:11:04 +0200
    From:        Christoph Badura <bad%bsd.de@localhost>
    Message-ID:  <20250418151104.GA15349%irregular-apocalypse.k.bsd.de@localhost>


  | You meant the -e flag to checkflist.

Yes.

  | But that does suppress all the warning about the extra files.

Huh?   Most of my recent builds have been into an empty or at least
unconfigured DESTDIR, so that -e (to checkflist) hasn't been needed,
but I did a random sampling of some older build logs I have, and
found this one:

=================================================  Mon Jan 27 05:28:47 +07 2025
===> build.sh command:    build.sh -x -j20 -V MKDEBUG=yes -V MKDEBUGLIB=yes -V MKKDEBUG=yes -V CHECKFLIST_FLAGS=-e -m amd64 -D /release/testing/amd64 -O /usr/obj/testing/amd64 -R /local/snap/20250126-testing-10.99.12-amd64 -T /usr/obj/testing/tools -X /readonly/release/testing/src/xsrc -u release

The first line (====...) is from my script (I think anyway, it has been a
long time since I did anything in that area), the next is from build.sh and
shows the args passed to it, including the -V CHECKFLIST_FLAGS=-e

I'll omit the following 167104 lines of build.sh (mostly make of course)
output, until we get to:

checkflist ===> distrib/sets
--- check_DESTDIR ---
--- sanitise_METALOG ---
--- checkflist ---
#   execute  checkflist
cd /readonly/release/testing/src/distrib/sets &&  DESTDIR=/release/testing/amd64  MACHINE=amd64  MAC
HINE_ARCH=x86_64  AWK=/usr/obj/testing/tools/bin/nbawk  CKSUM=/usr/obj/testing/tools/bin/nbcksum  DB
=/usr/obj/testing/tools/bin/nbdb  EGREP=/usr/obj/testing/tools/bin/nbgrep\ -E  HOST_SH=/bin/sh  MAKE
=/usr/obj/testing/tools/bin/nbmake  MKTEMP=/usr/obj/testing/tools/bin/nbmktemp  MTREE=/usr/obj/testi
ng/tools/bin/nbmtree  PAX=/usr/obj/testing/tools/bin/nbpax  COMPRESS_PROGRAM=/usr/obj/testing/tools/
bin/nbxz  GZIP=-n  XZ_OPT=-9  TAR_SUFF=tar.xz  PKG_CREATE=/usr/obj/testing/tools/bin/nbpkg_create  S
ED=/usr/obj/testing/tools/bin/nbsed  TSORT=/usr/obj/testing/tools/bin/nbtsort\ -q  /bin/sh /readonly
/release/testing/src/distrib/sets/checkflist  -L base,x -e
=======  303 extra files in DESTDIR  =========
Files in DESTDIR but missing from flist.
File is obsolete or flist is out of date ?
This is non-fatal, due to '-e' option.
------------------------------------------
./bin/sh.core
./dev/agpgart

(I'll omit the other 301 of them, but it includes stuff like /home/kre
and various junk in /tmp and /var from when I've been running tests).
The sh.core is a little embarrassing, but I do work on sh, and sometimes
that work doesn't work as intended...

=========  end of 303 extra files  ===========
======  1 missing files in DESTDIR  ========
Files in flist but missing from DESTDIR.
File wasn't installed ?
------------------------------------------
./usr/libdata/debug/usr/tests/lib/libexecinfo/t_backtrace_sandboc.debug
========  end of 1 missing files  ==========

That last one was probably what that build was done for, that build
ended immediately after this (the missing file is a bug).

There's another build immediately after:

=================================================  Mon Jan 27 05:41:50 +07 2025
===> build.sh command:    build.sh -x -j20 -V MKDEBUG=yes -V MKDEBUGLIB=yes -V MKKDEBUG=yes -V CHECK
FLIST_FLAGS=-e -m amd64 -D /release/testing/amd64 -O /usr/obj/testing/amd64 -R /local/snap/20250126-
testing-10.99.12-amd64 -T /usr/obj/testing/tools -X /readonly/release/testing/src/xsrc -u release
[....]
=======  302 extra files in DESTDIR  =========
Files in DESTDIR but missing from flist.
File is obsolete or flist is out of date ?
This is non-fatal, due to '-e' option.
------------------------------------------
./bin/sh.core
[....]
=========  end of 302 extra files  ===========

make distribution started at:  Mon Jan 27 05:41:51 +07 2025
make distribution finished at: Mon Jan 27 05:47:03 +07 2025
release ===> etc        (with: DISTRIBUTION_DONE=1)

(and so it continues to finish making the release).   The change from
303 extra files to 302 was because the problem in the first build was
a typo in the sets list, the file was installed, under its correct name,
the sets list was corrected (s/sandboc/sandbox/) before this build, and
all was good.

But what was it you were claiming about "suppress all the warning about
the extra files"?   I couldn't find where it did that.

  | Plus the feedback of what extra files were exempted by the extra
  | prefixes so that you can check if that matches what you intended.

If you really feel a need to post-process that so that the output
doesn't contain all the files in /tmp /var/log /home/kre (just /home
I think) then fine, but I don't see the need, it looks to be "like a very
quick hack for a very specific problem that was easy to solve with existing
machinery." to quote a message on a slightly different topic from the list,
that, I believe, you sent.   A simple sed script (or awk, perl, probably pure
sh) on the checkflist output could easily drop all lines after an added
extra directory that checkflist reports exists, or whatever else someone,
who needs that, requires in their particular case.

  | I do note that you do use options to tell build.sh to set various variables
  | used by the build system.

Yes, that's what I said.

  | Those are not available in your scripts outside build.sh.

Huh?

  | As I mentioned that is one reason for my change.

Again: "a very quick hack ... easy to solve with existing machinery."

  | Variables like
  | NETBSDSRCDIR,

That's "." (or $PWD) when the script is run, you need to know that
already to run build.sh at all, as that's where build.sh lives.  No
need to export that from from the make system.

  | DESTDIR,
  | RELEASEDIR,
  | MACHINE

Did you miss:
>                         -m "${ARCH}"                    \
>                         -D "${DEST}"                    \
>                         -R "${SNAP}"                    \

which is exactly those three, the names I use are different, but
it's my script, I get to use whatever names I like (I like my 4
char words...) - so can you.  Hacking the make system isn't needed
for any of this.

  | and MACHINE_ARCH etc.

That would be -a to build.sh if I ever needed it for anything, so
far I haven't (but I don't build all ports) so I haven't bothered,
but it would be trivial to add.   I can't comment on "etc" without
knowing what that is, or why it would be needed.

  | I'm actually not clear if you are trying to discuss the change
  | I've proposed or how you organize your build scripts.

The former, as I believe it is, as you have described other (actually
more useful, potentially anyway) proposed changes "a hack" and should
not be added to the build system.   I used my script (one of them anyway)
merely to illustrate why it isn't needed.   I also don't see much in
the way of support for your proposal (in fact, I've seen none).

kre