pkg/59032: [macOS] pkgsrc python312 doesn't build, system expat header newer than library used

[coypu%sdf.org@localhost]

>Number:         59032
>Category:       pkg
>Synopsis:       [macOS] pkgsrc python312 doesn't build, system expat header newer than library used
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Jan 26 09:10:00 +0000 2025
>Originator:     coypu
>Release:        pkgsrc-current as of 20250126
>Organization:
>Environment:
Darwin H2X9X4V3CT 23.6.0 Darwin Kernel Version 23.6.0: Wed Jul 31 20:48:04 PDT 2024; root:xnu-10063.141.1.700.5~1/RELEASE_ARM64_T6030 arm64

>Description:
While building lang/python312, it fails due to an unexpected missing symbol in system expat.

clang -bundle -undefined dynamic_lookup -L/Users/maya/pkgsrc-20250126/pkg/lib -L/usr/lib -L/Users/maya/pkgsrc-20250126/pkg/lib -L/usr/lib    Modules/_elementtree.o   -o Modules/_elementtree.so
[ERROR] _elementtree failed to import: dlopen(/Users/maya/pkgsrc-20250126/pkgsrc/lang/python312/work/Python-3.12.8/build/lib.macosx-14.7-arm64-3.12/pyexpat.so, 0x0002): Symbol not found: _XML_SetReparseDeferralEnabled
  Referenced from: <88C3DA05-DF93-33DD-AC79-02CA50A36F9B> /Users/maya/pkgsrc-20250126/pkgsrc/lang/python312/work/Python-3.12.8/Modules/pyexpat.so
  Expected in:     <F9FF65D5-8636-3FDE-B6E7-EA98E40F4FFC> /usr/lib/libexpat.1.dylib
[ERROR] pyexpat failed to import: dlopen(/Users/maya/pkgsrc-20250126/pkgsrc/lang/python312/work/Python-3.12.8/build/lib.macosx-14.7-arm64-3.12/pyexpat.so, 0x0002): Symbol not found: _XML_SetReparseDeferralEnabled
  Referenced from: <88C3DA05-DF93-33DD-AC79-02CA50A36F9B> /Users/maya/pkgsrc-20250126/pkgsrc/lang/python312/work/Python-3.12.8/Modules/pyexpat.so
  Expected in:     <F9FF65D5-8636-3FDE-B6E7-EA98E40F4FFC> /usr/lib/libexpat.1.dylib
Following modules built successfully but were removed because they could not be imported:
_elementtree          pyexpat                                  

Then the package fails to install:
/usr/bin/install -c -o maya -g staff -m 755 Modules/pyexpat.so /Users/maya/pkgsrc-20250126/pkg/lib/python3.12/lib-dynload/pyexpat.so
install: Modules/pyexpat.so: No such file or directory
gmake: *** [Makefile:2087: sharedinstall] Error 71
*** Error code 2

Stop.
bmake[3]: stopped making "package-install" in /Users/maya/pkgsrc-20250126/pkgsrc/lang/python312


this symbol is in expat 2.6.0:
> diff -u /Library/Developer/CommandLineTools/SDKs/MacOSX14.5.sdk/usr/include/expat.h  /Library/Developer/CommandLineTools/SDKs/MacOSX15.0.sdk/usr/include/expat.h | grep -C2 XML_SetReparseDeferralEnabled
+/* Added in Expat 2.6.0. */
+XMLPARSEAPI(XML_Bool)
+XML_SetReparseDeferralEnabled(XML_Parser parser, XML_Bool enabled);
+
 /* Expat follows the semantic versioning convention.


The code in python is conditional on having the correct version header, so it should just work:

#if XML_COMBINED_VERSION >= 20600
    XML_SetReparseDeferralEnabled(self->itself, enabled ? XML_TRUE : XML_FALSE);
    self->reparse_deferral_enabled = (bool)enabled;
#endif         
    Py_RETURN_NONE;       



But macos uses a header newer (15.0) than the system library (14.x)
OSX_SDK_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk and

> ls -l /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
lrwxr-xr-x  1 root  wheel  14 Sep 25 14:04 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk@ -> MacOSX15.0.sdk

So the header contains a definition that we can't use.

(Also, macOS doesn't let you inspect the contents of the system libraries, at least on my very corporate work laptop, so it's hard to diagnose what's happening.)
>How-To-Repeat:

>Fix:
jperkin mentions building macos packages with --prefer-pkgsrc=yes, and recommends it to avoid issues.

I suspect we don't have enough macOS know-how and users to maintain good state of conditions while using system libraries at the moment and I think this should be a default.