tech-userlevel archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[PATCH] Transitive LIBDPLIBS, PROGDPLIBS
[bcc tech-userlevel, followups to tech-toolchain]
tl;dr: Teach bsd.prog.mk and bsd.lib.mk to carry transitive library
dependencies through LIBDPLIBS/PROGDPLIBS. OK?
### BACKGROUND
If a program or library has to be linked against a library, how do you
express this with bsd.prog.mk or bsd.lib.mk?
Say it's a program that uses libsqlite3. You might start with:
LDADD+= -lsqlite3
DPADD+= ${LIBSQLITE3}
This simply adds -lsqlite3 to the linker command line, and asks to
relink if ${LIBSQLITE3} (i.e., ${DESTDIR}/usr/lib/libsqlite3.a) has
changed.
Except that's not enough on static builds (like sun2), because
libsqlite3 uses libm, and static libraries don't record those
transitive dependencies, so you have to do:
LDADD+= -lsqlite3 -lm
DPADD+= ${LIBSQLITE3} ${LIBM}
We have various programs with deeper dependencies, like anything that
uses krb5 via gssapi, which is so inconvenient to maintain that we've
baked definitions like
LIBKRB5_LDADD+= -lkrb5 -lcom_err \
-lhx509 -lcrypto -lasn1 \
-lwind -lheimbase -lcom_err -lroken \
-lcrypt -lutil
LIBGSSAPI_LDADD+= -lgssapi -lheimntlm ${LIBKRB5_LDADD}
into bsd.prog.mk itself, and we have to update them from time to time
as the sun2 build breaks.
The LDADD/DPADD mechanism also works only for libraries that are
installed into DESTDIR. Private libraries which are not installed
already require the use of LIBDPLIBS/PROGDPLIBS to get
-L${OBJDIR}/lib/libfoo (or manual cd ${NETBSDSRCDIR}/lib/libfoo &&
${PRINTOBJDIR}, which should be eliminated anyway).
### USAGE
Using the mechanism proposed in this patch (not yet applied to
Makefiles throughout the tree -- to be done in separate changes),
libsqlite3's Makefile (using bsd.lib.mk) will have:
LIBDPLIBS+= m ${NETBSDSRCDIR}/lib/libm
And a program (using bsd.prog.mk) that needs to use libsqlite3 can
just have:
PROGDPLIBS+= sqlite3 ${NETBSDSRCDIR}/external/public-domain/sqlite3/lib
The syntax is the same as bsd.lib.mk and bsd.prog.mk already support
mainly for private libraries which don't get installed in DESTDIR.
This syntax will set up LDADD and DPADD appropriately so that:
(a) the necessary -L arguments are passed through (whether sqlite3 is
a private static library or a public shared library);
(b) the necessary -l arguments are passed through (just -lsqlite3 for
dynamic builds; -lsqlite3 -lm for static builds); and
(c) the necessary make(1) dependencies will be created so if
libsqlite3 is rebuilt -- or if its LIBDPLIBS changes -- then the
program will be rebuilt too.
The change is that LIBPDLIBS and PROGDPLIBS will bring in the
transitive dependencies, not just the direct dependencies.
Similarly, all the transitive LIBKRB5_LDADD and LIBGSSAPI_LDADD goop
can go away; programs that use libgssapi can just do:
PROGDPLIBS+= gssapi ${NETBSDSRCDIR}/crypto/external/bsd/heimdal/lib/libgssapi
No need to spell out krb5, com_err, hx509, crypto, asn1, wind,
heimbase, roken, crypt, util. This should substantially improve the
legibility and maintenance burden of program and library makefiles.
### MECHANISM
1. When a library is built with bsd.lib.mk, its LIBDPLIBS, and the
objdirs (obtained by cd ${dir} && ${PRINTOBJDIR}) of its
dependencies, are written out to a file ${LIB}.deps.mk. The file
will look something like this:
LIBDPLIBS.sqlite3= m /home/riastradh/netbsd/current/src/lib/libm
LIBDO.m= /home/riastradh/netbsd/current/src/../obj.amd64/lib/libm
2. When one of the libraries used in LIBDPLIBS has a ${LIB}.deps.mk
file, it is included to get the transitive dependencies as well as
their cached objdirs from the LIBDO.${dep} variables.
(a) The transitive dependencies are concatenated in order, and then
deduplicated in reverse order, so that the topological order of
dependencies is preserved but we avoid a combinatorial
explosion. For example, suppose we have the following
dependencies:
foo: bar baz quux
bar: baz
quux: baz
A library that depends on foo, bar, and quux will first get:
(direct) (via foo) (via bar) (via quux)
foo bar quux bar baz quux baz baz
The duplicates will be removed (underlined) with the _last_
instance of each duplicate kept:
foo bar quux bar baz quux baz baz
^^^ ^^^^ ^^^ ^^^
The resulting list of transitive dependencies is:
foo bar quux baz
This preserves the order of dependency relations as required by
ld(1).
(b) Caching the objdirs means that when running make in a library
directory, we only need run ${PRINTOBJDIR} subshells for each
of the _direct_ dependencies, not the transitive dependencies.
If the .deps.mk file has not been built yet, then bsd.lib.mk
will defer to transitive subshells as a convenience when
running make(1) in a library directory before its dependencies
have been built. This process passes a make(1) variable
_LIBDPLIBS_PATH to detect cycles so it won't loop.
The subshell fallback might not be worthwhile -- e.g., it will
still hurt when running make includes at the beginning. For
example, on my laptop, make(1) takes about 10sec to process it
all in crypto/external/bsd/heimdal/lib/libgssapi when none of
the .deps.mk files exist. To be considered.
3. When a program is built with bsd.prog.mk with PROGDPLIBSSTATIC=yes,
or MKPIC=no, any transitive dependencies of its PROGDPLIBS are
picked up from the .deps.mk files like LIBDPLIBS, or from subshells
if no .deps.mk files exist yet.
(No transitive dependencies are picked up by bsd.prog.mk for
dynamic builds -- they're not necessary because they're already
baked into the .so files.)
4. If LIBDO.foo is set to `_external', none of the transitive
dependency resolution logic is triggered for libfoo, and -lfoo is
assumed to be available in the system library search path.
### DISCUSSION
pkg-config. This overlaps in purpose a little bit with pkg-config .pc
files, but by using make(1) syntax in the .deps.mk files it's a
little more convenient to process in bsd.lib.mk.
Although I don't think it's worthwhile to pull pkg-config (or
pkgconf) into the toolchain, perhaps we should use the fruits of
these transitive dependency calculations to create the Libs.private
lines in .pc files.
Incremental rebuilds. Using LIBDO.foo=_external enables incremental
rebuilds with USETOOLS=no, e.g. in SA instructions. For example, if
you're rebuilding a program that uses libgssapi with USETOOLS=no,
you will need to add LIBDO.gssapi=_external to the command line if
you don't want to also rebuild Heimdal.
That's more than was needed before with the LDADD=-lfoo approach,
but it's a small price to pay for the reduced maintenance burden and
legibility of makefiles. Really we should provide automated binary
updates -- more on that later -- but for now, at least this doesn't
break the opportunity for incremental updates in SA instructions.
Error checking. The logic has some error checking, so if, e.g., you
specify a source directory that doesn't exist in LIBDPLIBS,
bsd.lib.mk will yell at you. Similarly, if you set an entry for foo
in PROGDPLIBS, and some other library sets an entry for foo in its
LIBDPLIBS, and the source directories for these entries don't match,
bsd.prog.mk will yell at you.
build_install. A side effect of this change -- once it is adopted
uniformly throughout the tree -- is that it will obviate the need
for build_install (which runs dependall;install in batches in
directories like lib/ so that if libfoo depends on libbar, it has
run dependall;install in libbar first before starting dependall in
libfoo) because the libraries can all be resolved through objdir
rather than through destdir, so we can just do dependall in
dependency order through the whole tree before install.
Future work. By writing down the source tree references, we could use
this to automatically discover (and cache) fine-grained
directory-by-directory dependencies in, e.g., lib/Makefile, rather
than manually maintaining build_install groups separated by .WAIT
barriers. I expect this is an opportunity to substantially improve
parallelism of the library build.
Thoughts?
# HG changeset patch
# User Taylor R Campbell <riastradh%NetBSD.org@localhost>
# Date 1732120723 0
# Wed Nov 20 16:38:43 2024 +0000
# Branch trunk
# Node ID 46b3f4faf698dfd8bb58c02980f1d9fc326df5cd
# Parent 34daad9d3eef41eb9139db8022ea7b0dbae3aa05
# EXP-Topic riastradh-20241114-transitivedplibs
bsd.lib.mk, bsd.prog.mk: Carry transitive library dependencies.
For MKPIC=no builds:
1. Make bsd.lib.mk store LIBDPLIBS in ${.OBJDIR}/${LIB}.deps.mk so
downstream users can get at it.
2. Automatically add upstream .deps.mk dependencies to LIBDPLIBS with
bsd.lib.mk or PROGDPLIBS with bsd.prog.mk.
This way, the .deps.mk file always has all the transitive
dependencies, which are made available when linking programs with
PROGDPLIBS, so, e.g., if libarchive needs libz, programs that
directly need only libarchive can declare
PROGDPLIBS+= archive ${NETBSDSRCDIR}/external/bsd/libarcive/lib/libarchive
and they will be linked with -larchive -lz.
Duplicates are eliminated in the cached LIBDPLIBS, preserving
topological order, to avoid combinatorial explosion of library
dependencies which overrun linker command-line argument lengths.
Duplicate library paths are checked to avoid accidentally trying to
get `-lfoo' from two different directories. The check uses string
equality first so it doesn't add overhead for all the
${NETBSDSRCDIR}/... paths, but if that fails, it forks a cd && pwd
subshell to verify that ${.CURDIR}/... and ${.PARSEDIR}/... paths
match. This way we make sure when linking `-lfoo' directly and
transitively, it's for the same libfoo.
This logic works consistently with LIBDO.${lib}=_external. This way
it should be possible to incrementally build a program or library at
a time with fine-grained control over which libraries it links
against come from the host environment -- as in with USETOOLS=no,
e.g. to follow an SA's instructions -- and which libraries it needs
to get out of the build.
diff -r 34daad9d3eef -r 46b3f4faf698 share/mk/bsd.lib.mk
--- a/share/mk/bsd.lib.mk Thu Nov 14 03:02:25 2024 +0000
+++ b/share/mk/bsd.lib.mk Wed Nov 20 16:38:43 2024 +0000
@@ -70,25 +70,138 @@ PGFLAGS+= -fPIC
.endif
##### Libraries that this may depend upon.
-.if defined(LIBDPLIBS) && ${MKPIC} != "no" # {
+.if defined(LIBDPLIBS) # {
+# Make sure we're not in a dependency cycle.
+. if !empty(_LIBDPLIBS_PATH:M${LIB})
+. error lib${LIB}: dependency cycle: ${_LIBDPLIBS_PATH} ${LIB}
+. endif
+# Process each direct dependency ${_lib} in source ${_dir} to find:
+#
+# 1. its objdir (or "_external" if we're linking it from the host
+# environment rather than an internal library), cached in
+# LIBDO.${_lib}, and
+#
+# 2. if we're statically linking, its transitive dependencies, cached
+# in LIBDPLIBS.${_lib}.
+#
+# Put the list of dependencies (direct, for dynamic linking;
+# transitive, for static linking) into _LIBDPLIBS_ALL -- this may
+# contain duplicates; we will deduplicate later.
+_LIBDPLIBS_ALL= ${LIBDPLIBS}
. for _lib _dir in ${LIBDPLIBS}
. if !defined(LIBDO.${_lib})
+. if ${_dir} == "_external"
+LIBDO.${_lib}= _external
+. elif !exists(${_dir})
+. error lib${LIB}: dependency lib${_lib}: ${_dir} does not exist
+. else
LIBDO.${_lib}!= cd "${_dir}" && ${PRINTOBJDIR}
.MAKEOVERRIDES+=LIBDO.${_lib}
-. endif
-. if ${LIBDO.${_lib}} == "_external"
-LDADD+= -l${_lib}
-. else
-LDADD+= -L${LIBDO.${_lib}} -l${_lib}
-. if exists(${LIBDO.${_lib}}/lib${_lib}_pic.a)
-DPADD+= ${LIBDO.${_lib}}/lib${_lib}_pic.a
-. elif exists(${LIBDO.${_lib}}/lib${_lib}.so)
-DPADD+= ${LIBDO.${_lib}}/lib${_lib}.so
-. else
-DPADD+= ${LIBDO.${_lib}}/lib${_lib}.a
. endif
. endif
+# For static linking (of an internal library):
+#
+# 1. If we already have LIBDPLIBS.${_lib} cached, nothing to do.
+#
+# 2. If we can get the cached transitive dependencies out of
+# ${_lib}.deps.mk in the objdir, do that.
+#
+# 3. Otherwise, run a recursive make to get _LIBDPLIBS -- tracking the
+# path of library dependencies in _LIBDPLIBS_PATH so we can detect
+# cycles -- and save it in LIBDPLIBS.${_lib}.
+#
+# In any case, we get the transitive dependencies of ${_lib} in
+# LIBDPLIBS.${_lib}; add them to _LIBDPLIBS_ALL.
+. if ${LIBDO.${_lib}} != "_external"
+. if defined(LIBDPLIBS.${_lib})
+# nothing to do
+. elif exists(${LIBDO.${_lib}}/${_lib}.deps.mk)
+. include "${LIBDO.${_lib}}/${_lib}.deps.mk"
+. if !defined(LIBDPLIBS.${_lib})
+. error lib${LIB}: dependency lib${_lib}: ${LIBDO.${_lib}}/${_lib}.deps.mk failed to define LIBDPLIBS.${_lib}
+. endif
+${LIB}.deps.mk: ${LIBDO.${_lib}}/${_lib}.deps.mk
+. else
+LIBDPLIBS.${_lib}!= cd ${_dir} && \
+ ${MAKE} -v _LIBDPLIBS _LIBDPLIBS_PATH="${_LIBDPLIBS_PATH} ${LIB}"
+# Make sure LIBDO.${_lib0} is defined for every transitive dependency
+# ${_lib0}. Do this eagerly so we can save it through recursive makes.
+# (Not perfect: we don't gather the objdirs from recursive makes, but
+# at least we do save them from the top level into the recursive
+# makes.)
+. for _lib0 _dir0 in ${_LIBDPLIBS}
+. if !defined(LIBDO.${_lib0})
+. if ${_dir0} == "_external"
+LIBDO.${_lib0}= _external
+. elif !exists(${_dir0})
+. error lib${_lib}: dependency lib${_lib0}: ${_dir0} does not exist
+. else
+LIBDO.${_lib0}!= cd ${_dir0} && ${PRINTOBJDIR}
+.MAKEOVERRIDES+= LIBDO.${_lib0}
+. endif
+. endif
+. endfor
+. endif
+_LIBDPLIBS_ALL+= ${LIBDPLIBS.${_lib}}
+. endif
. endfor
+# Nix duplicates from back to front, gathering the results in
+# _LIBDPLIBS_DEDUPREV. For each library ${_lib} that we have already
+# seen, set _LIBDPLIBS.${_lib} to its directory. This way we preserve
+# the topological order of dependencies but avoid explosions of
+# dependencies in case of duplicates.
+. for _dir _lib in ${_LIBDPLIBS_ALL:[-1..1]}
+. if !defined(_LIBDPLIBS.${_lib})
+_LIBDPLIBS.${_lib}= ${_dir}
+_LIBDPLIBS_DEDUPREV+= ${_dir} ${_lib}
+# If we've already seen this library name, make sure the directory
+# matches. Check string equality first because it's cheap, but since
+# lots of makefiles use ${.CURDIR}/... or ${.PARSEDIR}/... paths for
+# this, if string equality fails, fall back to forking cd && pwd.
+. elif ${_LIBDPLIBS.${_lib}} != ${_dir} && \
+ (!exists(${_dir}) || \
+ ${:!cd ${_LIBDPLIBS.${_lib}} && pwd!} != ${:!cd ${_dir} && pwd!})
+. error lib${LIB}: dependency lib${_lib}: ${_LIBDPLIBS.${_lib}} != ${_dir}
+. endif
+. endfor
+# Reverse _LIBDPLIBS_DEDUPREV to yield deduplicated, topologically
+# sorted _LIBDPLIBS. The logic above, and bsd.prog.mk, will use this
+# variable for transitive dependencies downstream.
+_LIBDPLIBS= ${_LIBDPLIBS_DEDUPREV:[-1..1]}
+# Save transitive dependencies and cached objdir paths in
+# ${LIB}.deps.mk so downstream statically linked programs can use them.
+# (XXX Overlapping purpose with pkgconfig!)
+CLEANFILES+= ${LIB}.deps.mk ${LIB}.deps.mk.tmp
+DPADD+= ${LIB}.deps.mk
+realdepend: ${LIB}.deps.mk
+${LIB}.deps.mk:
+ ${_MKTARGET_CREATE}
+ rm -f ${.TARGET} ${.TARGET}.tmp
+ echo 'LIBDPLIBS.${LIB}= ${_LIBDPLIBS}' >>${.TARGET}.tmp
+. for _lib _dir in ${_LIBDPLIBS}
+ echo 'LIBDO.${_lib}= ${LIBDO.${_lib}}' >>${.TARGET}.tmp
+. endfor
+ ${MV} ${.TARGET}.tmp ${.TARGET}
+# Apply the dependencies to LDADD/DPADD if we're building shared
+# libraries (MKPIC=yes). Static libraries don't record transitive
+# dependency information -- we have to apply that when linking the
+# final executables via the ${LIB}.deps.mk files.
+. if ${MKPIC} != "no"
+. for _lib _dir in ${LIBDPLIBS}
+. if ${LIBDO.${_lib}} == "_external"
+LDADD+= -l${_lib}
+. else
+LDADD+= -L${LIBDO.${_lib}} -l${_lib}
+. if exists(${LIBDO.${_lib}}/lib${_lib}_pic.a)
+DPADD+= ${LIBDO.${_lib}}/lib${_lib}_pic.a
+. elif exists(${LIBDO.${_lib}}/lib${_lib}.so)
+DPADD+= ${LIBDO.${_lib}}/lib${_lib}.so
+. else
+DPADD+= ${LIBDO.${_lib}}/lib${_lib}.a
+. endif
+. endif
+. endfor
+. endif
.endif # }
##### Build and install rules
diff -r 34daad9d3eef -r 46b3f4faf698 share/mk/bsd.prog.mk
--- a/share/mk/bsd.prog.mk Thu Nov 14 03:02:25 2024 +0000
+++ b/share/mk/bsd.prog.mk Wed Nov 20 16:38:43 2024 +0000
@@ -384,12 +384,103 @@ PROGS= ${PROG}
##### Libraries that this may depend upon.
.if defined(PROGDPLIBS) # {
+# Process each direct dependency ${_lib} in source ${_dir} to find:
+#
+# 1. its objdir (or "_external" if we're linking it from the host
+# environment rather than an internal library), cached in
+# LIBDO.${_lib}, and
+#
+# 2. if we're statically linking, its transitive dependencies, cached
+# in LIBDPLIBS.${_lib}.
+#
+# Put the list of dependencies (direct, for dynamic linking;
+# transitive, for static linking) into _PROGDPLIBS_ALL -- this may
+# contain duplicates; we will deduplicate later.
+_PROGDPLIBS_ALL=${PROGDPLIBS}
. for _lib _dir in ${PROGDPLIBS}
. if !defined(LIBDO.${_lib})
+. if ${_dir} == "_external"
+LIBDO.${_lib}= _external
+. elif !exists(${_dir})
+. error dependency lib${_lib}: ${_dir} does not exist
+. else
LIBDO.${_lib}!= cd "${_dir}" && ${PRINTOBJDIR}
.MAKEOVERRIDES+=LIBDO.${_lib}
+. endif
. endif
-. if defined(PROGDPLIBSSTATIC)
+# For static linking:
+#
+# 1. If we already have LIBDPLIBS.${_lib} cached, nothing to do.
+#
+# 2. If we can get the cached transitive dependencies out of
+# ${_lib}.deps.mk in the objdir, do that.
+#
+# 3. Otherwise, run a recursive make to get _LIBDPLIBS -- tracking the
+# path of library dependencies in _LIBDPLIBS_PATH so we can detect
+# cycles -- and save it in LIBDPLIBS.${_lib}.
+#
+# In any case, we get the transitive dependencies of ${_lib} in
+# LIBDPLIBS.${_lib}; add them to _PROGDPLIBS_ALL.
+. if defined(PROGDPLIBSSTATIC) || ${MKPIC} == "no"
+. if defined(LIBDPLIBS.${_lib}) || ${LIBDO.${_lib}:U} == "_external"
+# nothing to do
+. elif exists(${LIBDO.${_lib}}/${_lib}.deps.mk)
+. include "${LIBDO.${_lib}}/${_lib}.deps.mk"
+. if !defined(LIBDPLIBS.${_lib})
+. error dependency lib${_lib}: ${LIBDO.${_lib}}/${_lib}.deps.mk failed to define LIBDPLIBS.${_lib}
+. endif
+DPADD+= ${LIBDO.${_lib}}/${_lib}.deps.mk
+. else
+LIBDPLIBS.${_lib}!= cd ${_dir} && ${MAKE} -v _LIBDPLIBS
+# Make sure LIBDO.${_lib0} is defined for every transitive dependency
+# ${_lib0}. Do this eagerly so we can save it through recursive makes.
+# (Not perfect: we don't gather the objdirs from recursive makes, but
+# at least we do save them from the top level into the recursive
+# makes.)
+. for _lib0 _dir0 in ${LIBDPLIBS.${_lib}}
+. if !defined(LIBDO.${_lib0})
+. if ${_dir0} == "_external"
+LIBDO.${_lib0}= _external
+. elif !exists(${_dir0})
+. error lib${_lib}: dependency lib${_lib0}: ${_dir0} does not exist
+. else
+LIBDO.${_lib0}!= cd ${_dir0} && ${PRINTOBJDIR}
+.MAKEOVERRIDES+= LIBDO.${_lib0}
+. endif
+. endif
+. endfor
+. endif
+_PROGDPLIBS_ALL+= ${LIBDPLIBS.${_lib}}
+. endif
+. endfor
+# Nix duplicates from back to front, gathering the results in
+# _PROGDPLIBS_DEDUPREV. For each library ${_lib} that we have already
+# seen, set _PROGDPLIBS.${_lib} to its directory. This way we preserve
+# the topological order of dependencies but avoid explosions of
+# dependencies in case of duplicates.
+. for _dir _lib in ${_PROGDPLIBS_ALL:[-1..1]}
+. if !defined(_PROGDPLIBS.${_lib})
+_PROGDPLIBS.${_lib}= ${_dir}
+_PROGDPLIBS_DEDUPREV+= ${_dir} ${_lib}
+# If we've already seen this library name, make sure the directory
+# matches. Check string equality first because it's cheap, but since
+# lots of makefiles use ${.CURDIR}/... or ${.PARSEDIR}/... paths for
+# this, if string equality fails, fall back to forking cd && pwd.
+. elif ${_PROGDPLIBS.${_lib}} != ${_dir} && \
+ (!exists(${_dir}) || \
+ ${:!cd ${_PROGDPLIBS.${_lib}} && pwd!} != \
+ ${:!cd ${_dir} && pwd!})
+. error dependency lib${_lib}: ${_PROGDPLIBS.${_lib}} != ${_dir}
+. endif
+. endfor
+# Reverse _PROGDPLIBS_DEDUPREV to yield deduplicated, topologically
+# sorted _PROGDPLIBS.
+_PROGDPLIBS= ${_PROGDPLIBS_DEDUPREV:[-1..1]}
+# Add all the linker flags to LDADD and file dependencies to DPADD.
+. for _lib _dir in ${_PROGDPLIBS}
+. if ${LIBDO.${_lib}} == "_external"
+LDADD+= -l${_lib}
+. elif defined(PROGDPLIBSSTATIC)
DPADD+= ${LIBDO.${_lib}}/lib${_lib}.a
LDADD+= ${LIBDO.${_lib}}/lib${_lib}.a
. else
Home |
Main Index |
Thread Index |
Old Index