Re: pbulk: don't break on unresolvable dependencies

[Thomas Klausner <wiz%NetBSD.org@localhost>]

On Sat, Jan 04, 2025 at 05:31:21AM +0000, Taylor R Campbell wrote:
> > - remove ignore_missing_dependencies config setting because it's now
> >   unnecessary
> 
> Maybe we should make pbulk fail noisily, or at least warn noisily, if
> you set ignore_missing_dependencies, now that nothing uses it?

I've added a warning to this to the output.

I've also noticed that pbulk-scan's '-p' option is now always used
when not in incremental mode, so I made it the default in that case
and removed the option. I've also tried improving pbulk-scan's man
page. New diff attached.


Sample limited pbulk build for a local package py-ydist312 that
requires a py-xdist311 for Python 3.11 that doesn't support that
Python version:


# /usr/pkg_bulk/bin/bulkbuild
The 'ignore_missing_dependencies' configuration variable is obsolete and has no effect.
Scanning...
. 1/1
Resolving...
Scanning...
..... 5/5
Resolving...
Scanning...
....... 7/7
Resolving...
Scanning...
... 3/3
Resolving...
Scanning...
.. 2/2
Resolving...
Scanning...
. 1/1
Resolving...
Unresolvable dependencies found, marking packages depending on them as broken:
nih/py-xdist311
pbulk-resolve: No match found for dependency py311-xdist311-[0-9]* of package py311-ydist312-0.0.2
Building...
Initialisation complete.
[2/16] Starting build of  cwrappers-20220403
[2/16] Successfully built cwrappers-20220403
[3/16] Starting build of  checkperms-1.12
[3/16] Successfully built checkperms-1.12
[4/16] Starting build of  m4-1.4.19nb1
[5/16] Starting build of  gmake-4.4.1
[5/16] Successfully built m4-1.4.19nb1
[6/16] Starting build of  libtool-base-2.4.7nb1
[6/16] Successfully built gmake-4.4.1
[7/16] Starting build of  mpdecimal-4.0.0
[7/16] Successfully built libtool-base-2.4.7nb1
[7/16] Successfully built mpdecimal-4.0.0
[8/16] Starting build of  pkgconf-2.3.0
[9/16] Starting build of  sqlite3-3.47.2nb1
[9/16] Successfully built pkgconf-2.3.0
[9/16] Successfully built sqlite3-3.47.2nb1
[10/16] Starting build of  libuuid-2.40.2
[11/16] Starting build of  libffi-3.4.6
[11/16] Successfully built libuuid-2.40.2
[11/16] Successfully built libffi-3.4.6
[12/16] Starting build of  readline-8.2nb2
[12/16] Successfully built readline-8.2nb2
[13/16] Starting build of  python313-3.13.1nb3
[14/16] Starting build of  python311-3.11.11nb1
[14/16] Successfully built python313-3.13.1nb3
[15/16] Starting build of  py313-xdist311-0.0.2
[15/16] Successfully built python311-3.11.11nb1
[15/16] Failed to build    py313-xdist311-0.0.2
Building pkg_summary...
Building SHA512...
Build reports...
Sending report mail...
Uploading report...
Uploading packages...


Sample build with /usr/pkgsrc/Makefile just descending in mbone/ (thus
missing all required dependencies):

# /usr/pkg_bulk/bin/bulkbuild
The 'ignore_missing_dependencies' configuration variable is obsolete and has no effect.
Scanning...
............. 13/13
Resolving...
Building...
Initialisation complete.
Building pkg_summary...
Building SHA512...
Build reports...
Sending report mail...
Uploading report...
Uploading packages...

I'll attach the report.html generated by that last bulk build.


I'll commit the new version in a week if I get no further feedback.
 Thomas

Title: pkgsrc bulk build for NetBSD 10.99.12/x86_64 from 2025-01-06 19:40

pkgsrc bulk build for NetBSD 10.99.12/x86_64

Build start: 2025-01-06 19:40

Build end: 2025-01-06 19:40

---

Total number of packages:	12
Successfully built:	0
Failed build:	0
Depending on failed package:	0
Explicitly broken or masked:	12
Depending on masked package:	0

---

All unsuccessful builds

Location	Package	Breaks	Maintainer	Status	Build log
mbone/beacon	beacon-1.3nb15		pettai%NetBSD.org@localhost	prefailed
	could not resolve dependency perl>=5.40.0
mbone/common-mml	common-mml-1.2.16nb4		psham%cs.nmsu.edu@localhost	prefailed
	could not resolve dependency libtool-base>=2.4.2nb9
mbone/dbeacon	dbeacon-0.3.9.1nb5		pkgsrc-users%NetBSD.org@localhost	prefailed
	could not resolve dependency gmake>=3.81
mbone/mdd	mdd-20230428		manu%NetBSD.org@localhost	prefailed
	could not resolve dependency cwrappers>=20150314
mbone/mdp	mdp-1.9a5nb1		pkgsrc-users%NetBSD.org@localhost	prefailed
	could not resolve dependency gmake>=3.81
mbone/rat	rat-3.0.28nb12		pkgsrc-users%NetBSD.org@localhost	prefailed
	could not resolve dependency gmake>=3.81
mbone/rtptools	rtptools-1.22		pkgsrc-users%NetBSD.org@localhost	prefailed
	could not resolve dependency automake>=1.9
mbone/sapserver	miniSAPserver-0.2.4		pkgsrc-users%NetBSD.org@localhost	prefailed
	could not resolve dependency cwrappers>=20150314
mbone/vat	vat-4.0b2nb12		pkgsrc-users%NetBSD.org@localhost	prefailed
	could not resolve dependency cwrappers>=20150314
mbone/vic	vic-2.8nb12		is%NetBSD.org@localhost	prefailed
	could not resolve dependency cwrappers>=20150314
mbone/vic-devel	vic-devel-2.8.1.4.0beta.4337nb21		sborrill%NetBSD.org@localhost	prefailed
	could not resolve dependency cwrappers>=20150314
mbone/wbd	wbd-1.0p4nb11		pkgsrc-users%NetBSD.org@localhost	prefailed
	could not resolve dependency cwrappers>=20150314

Index: files/pbulk/pbulk.conf
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pbulk/files/pbulk/pbulk.conf,v
retrieving revision 1.27
diff -u -r1.27 pbulk.conf
--- files/pbulk/pbulk.conf	12 Oct 2024 10:43:52 -0000	1.27
+++ files/pbulk/pbulk.conf	6 Jan 2025 18:43:54 -0000
@@ -44,11 +44,6 @@
 # The file is a newline separated list of package locations (e.g. lang/perl5).
 #limited_list=/limited_list
 
-# Optionally ignore unresolvable dependencies for a full build (e.g.
-# a build without limited_list set).  Default behavior is to bail out.
-#
-ignore_missing_dependencies=no
-
 # If yes, consider a package up-to-date, if the dependency list matches
 # the existing binary package and the recorded RCS IDs match the pkgsrc
 # tree. Otherwise, additionally require that the package is not older
Index: files/pbulk/presolve/pbulk-resolve.1
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pbulk/files/pbulk/presolve/pbulk-resolve.1,v
retrieving revision 1.1.1.1
diff -u -r1.1.1.1 pbulk-resolve.1
--- files/pbulk/presolve/pbulk-resolve.1	19 Jun 2007 19:49:57 -0000	1.1.1.1
+++ files/pbulk/presolve/pbulk-resolve.1	6 Jan 2025 18:43:54 -0000
@@ -24,7 +24,7 @@
 .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 .\" POSSIBILITY OF SUCH DAMAGE.
 .\"
-.Dd June 12, 2007
+.Dd January 6, 2025
 .Dt PBULK-RESOLVE 1
 .Os
 .Sh NAME
@@ -43,9 +43,17 @@
 .Xr pbulk-scan 1
 saved in
 .Ar input .
+Resolving a dependency means converting a pattern of the form
+.Dq sqlite3-[0-9]*:../../databases/sqlite3
+to the actual
+.Dv PKGNAME
+of the sqlite3 package found in
+.Ar input ,
+e.g.,
+.Dq sqlite3-3.47.2nb1 .
 .Pp
 Supported options are:
-.Bl -tag -offset indent
+.Bl -tag -offset indent -width 12n
 .It Fl i Ar missing
 Enter incremental mode.
 For incremental mode, the package in
@@ -55,8 +63,6 @@
 dependencies.
 The location of each unresolved dependency is written to
 .Ar missing .
-In normal mode, unresolvable dependencies are printed and the
-program exits with an error.
 .It Fl v
 If
 .Fl v
@@ -70,8 +76,8 @@
 .El
 .Sh EXIT STATUS
 .Nm
-exits with return value 1 if an error occurred, or 0 if all
-dependencies have been resolved successfully.
+exits with return value 0.
+.Pp
 In incremental mode,
 .Ar missing
 contains all correct, but unresolvable dependencies.
Index: files/pbulk/presolve/presolve.c
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pbulk/files/pbulk/presolve/presolve.c,v
retrieving revision 1.4
diff -u -r1.4 presolve.c
--- files/pbulk/presolve/presolve.c	16 Sep 2008 18:21:30 -0000	1.4
+++ files/pbulk/presolve/presolve.c	6 Jan 2025 18:43:54 -0000
@@ -46,13 +46,13 @@
 
 #include "pbulk.h"
 
-static int partial, verbosity;
+static int verbosity;
 static FILE *incremental = NULL;
 
 static void
 usage(void)
 {
-	(void)fprintf(stderr, "usage: pbulk-resolve [ -pv ] [ -i <missing> ] <pscan output> [ ... ]\n");
+	(void)fprintf(stderr, "usage: pbulk-resolve [-v] [-i missing] <pscan output> [ ... ]\n");
 	exit(1);
 }
 
@@ -63,7 +63,7 @@
 	char *depends;
 	char *pkglocation;
 	int active;
-	int broken; /* Entry has missing dependencies */
+	char *broken; /* Entry has missing dependencies */
 	const char *begin;
 	const char *end;
 	SLIST_ENTRY(pkg_entry) hash_link;
@@ -95,9 +95,6 @@
 			if ((incremental = fopen(optarg, "w")) == NULL)
 				err(1, "Cannot open output file");
 			break;
-		case 'p':
-			++partial;
-			break;
 		case 'v':
 			++verbosity;
 			break;
@@ -109,10 +106,7 @@
 	argc -= optind;
 	argv += optind;
 
-	if (argc == 0 || (incremental == NULL && argc > 1))
-		usage();
-
-	if (partial && incremental != NULL)
+	if (argc == 0)
 		usage();
 
 	read_entries(argv[0], 1);
@@ -286,6 +280,10 @@
 
 		if (best_match == NULL) {
 			ret = 1;
+			if (incremental == NULL && pkg->broken == NULL) {
+				/* only keep first broken dependency */
+				pkg->broken = xstrndup(pattern_begin, pattern_end - pattern_begin);
+			}
 			continue;
 		}
 
@@ -308,10 +306,6 @@
 	if (ret == 1) {
 		free(pkg->depends);
 		pkg->depends = NULL;
-		if (incremental != NULL || partial)
-			return 0;
-		else
-			return 1;
 	}
 	return 0;
 }
@@ -376,6 +370,7 @@
 		pkgs[len_pkgs].begin = input_iter;
 		pkgs[len_pkgs].end = pbulk_item_end(input_iter);
 		pkgs[len_pkgs].depends = NULL;
+		pkgs[len_pkgs].broken = NULL;
 		if (pkgs[len_pkgs].end == NULL)
 			errx(1, "Invalid input");
 		input_iter = pkgs[len_pkgs].end;
@@ -397,15 +392,35 @@
 		errx(1, "Invalid input");
 }
 
+#define PKG_FAIL_REASON "PKG_FAIL_REASON="
 static void
 write_entries(void)
 {
 	size_t i;
+	const char *line, *line_end;
 
 	for (i = 0; i < len_pkgs; ++i) {
 		if (pkgs[i].active == 0)
 			continue;
-		(void)fwrite(pkgs[i].begin, 1, pkgs[i].end - pkgs[i].begin, stdout);
+		/* if package is ok, just print existing entry */
+		if (pkgs[i].broken == NULL) {
+			(void)fwrite(pkgs[i].begin, 1, pkgs[i].end - pkgs[i].begin, stdout);
+		} else {
+			/* otherwise, replace PKG_FAIL_REASON line with reason for brokenness */
+			line = strstr(pkgs[i].begin, PKG_FAIL_REASON);
+			if (line == NULL) {
+				(void)fwrite(pkgs[i].begin, 1, pkgs[i].end -
+					     pkgs[i].begin, stdout); continue;
+			}
+			(void)fwrite(pkgs[i].begin, 1, line - pkgs[i].begin,
+			    stdout);
+			line_end = strchr(line, '\n');
+			(void)printf(PKG_FAIL_REASON
+			    "\"could not resolve dependency \"%s\"\"\n",
+				     pkgs[i].broken);
+			(void)fwrite(line_end + 1,
+				     1, pkgs[i].end - (line_end + 1), stdout);
+		}
 		if (pkgs[i].depends != NULL)
 			(void)printf("DEPENDS=%s\n", pkgs[i].depends);
 	}
Index: files/pbulk/scripts/scan
===================================================================
RCS file: /cvsroot/pkgsrc/pkgtools/pbulk/files/pbulk/scripts/scan,v
retrieving revision 1.10
diff -u -r1.10 scan
--- files/pbulk/scripts/scan	8 Jan 2020 15:12:01 -0000	1.10
+++ files/pbulk/scripts/scan	6 Jan 2025 18:43:54 -0000
@@ -47,6 +47,10 @@
 	extra_pscan_args=""
 fi
 
+if [ -n "${ignore_missing_dependencies}" ]; then
+	echo "The 'ignore_missing_dependencies' configuration variable is obsolete and has no effect." >&2
+fi
+
 if [ -z "${limited_list}" ]; then
 	echo "Scanning..."
 	case "${master_mode}" in
@@ -62,20 +66,7 @@
 		;;
 	esac
 	echo "Resolving..."
-	if ! ${presolve} -v ${loc}/pscan > ${loc}/presolve 2> ${loc}/presolve-err.log; then
-		echo "Global dependency resolution failed, check ${loc}/presolve-err.log for details" >&2
-		case "${ignore_missing_dependencies}" in
-		[nN][oO])
-			exit 1
-			;;
-		[yY][eE][sS])
-			${presolve} -p -v ${loc}/pscan > ${loc}/presolve 2> /dev/null
-			;;
-		*)
-			echo "Unsupported value for ignore_missing_dependencies."
-			;;
-		esac
-	fi
+	${presolve} -v ${loc}/pscan > ${loc}/presolve 2> ${loc}/presolve-err.log
 else
 	initial=1
 	mkdir -p ${loc}
@@ -108,9 +99,10 @@
 	done
 
 	if [ -s ${loc}/missing ]; then
-		echo "Unresolvable dependencies found, exiting:"
+		echo "Unresolvable dependencies found, marking packages depending on them as broken:"
 		cat ${loc}/missing
-		exit 1
+		${presolve} -v ${loc}/presolve ${loc}/pscan > ${loc}/presolve.new
+		mv ${loc}/presolve.new ${loc}/presolve
 	fi
 fi
 date '+SCAN_END_EPOCHE=%s' >> ${loc}/status