Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles that include devel/zlib/buildlink3.mk

["Lai, Peter PW" <peter.lai2%prattwhitney.com@localhost>]

> -----Original Message-----
> From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
> Sent: Friday, August 14, 2020 5:23 PM
> To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>; pkgsrc-users%netbsd.org@localhost
> Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles
> that include devel/zlib/buildlink3.mk
> 
> On 2020-08-10 12:05, Lai, Peter PW wrote:
> >> -----Original Message-----
> >> From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
> >> Sent: Monday, August 10, 2020 10:00 AM
> >> To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>;
> >> pkgsrc-users%netbsd.org@localhost
> >> Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash
> >> for Makefiles that include devel/zlib/buildlink3.mk
> >>
> >> On 2020-08-10 05:50, Lai, Peter PW wrote:
> >>> Environment:
> >>>
> >>> * RHEL 7.7 (Centos 7 for those playing at home I guess) in Azure.
> >>> * Toolchain: Software collection (SCL) devtoolset-9
> >>>
> >>> * Yum installs native zlib-devel as a dependency for openssl-devel
> >>>
> >>> * openssl-devel was installed in order to support PREFER_NATIVE=
> >> openssl
> >>> Observations:
> >>>
> >>> In this environment because zlib-devel is installed, bootstrap by
> >>> default
> >> picks up native zlib (/usr/include/zlib.h). When devel/libxml2 is
> >> then built, it is built against native zlib (libz.so.1 => /lib64/libz.so.1).
> >>> Next, some packages that also include
> >>> "../../devel/zlib/buildlink3.mk" such
> >> nginx is built, it pulls in devel/zlib. This is weird because RHEL 7
> >> includes zlib
> >> 1.2.7 which satisfies zlib>=1.2.3 in buildlink3.
> >>> What happens after is that libxml2 and downstream libxml2
> >>> dependencies
> >> (e.g. databases/postgresql) break because they see the pkgsrc zlib first:
> >>> /mnt/datadisk01/pkg/bin/postgres: /mnt/datadisk01/pkg/lib/libz.so.1:
> >>> no version information available (required by
> >>> /mnt/datadisk01/pkg/lib/libxml2.so.2)
> >>>
> >>> # ldd /mnt/datadisk01/pkg/lib/libxml2.so.2
> >>> /mnt/datadisk01/pkg/lib/libxml2.so.2:
> >>> /mnt/datadisk01/pkg/lib/libz.so.1: no version information available
> >>> (required by /mnt/datadisk01/pkg/lib/libxml2.so.2)
> >>> /mnt/datadisk01/pkg/lib/libxml2.so.2:
> >>> /mnt/datadisk01/pkg/lib/libz.so.1: no version information available
> >>> (required by /mnt/datadisk01/pkg/lib/libxml2.so.2)
> >>>
> >>> I worked around this by adding zlib to PREFER_NATIVE when
> >>> re-building
> >> nginx, but is there a better way to resolve this? This looks somewhat
> >> related to the continuing "pkg_install fails when archivers/xz is
> >> installed" thread because RPATH manipulation could be involved, but I
> >> am opening a separate thread in case it is not.
> >> This does look like the same issue that we've been discussing
> >> regarding xz and it's a great example of the kind of problems that
> >> are avoided by using PREFER_PKGSRC=yes (and/or PREFER_NATIVE=no).
> >>
> >> What's your motive for using native openssl instead of pkgsrc?  The
> >> only issue I've seen with the latter is the need to manually install
> >> security/mozilla-rootcerts* or equivalent to make curl, etc. work with
> https.
> >>
> >> If there's some other disadvantage to pkgsrc openssl, maybe we should
> >> try to correct it.
> > The RHEL version of OpenSSL is backported/patched 1.0.2k with the FIPS
> module compiled (and validated through vendor affirmed basis as per
> https://urldefense.com/v3/__https://www.openssl.org/docs/fips.html__;!!
> MvWE!V8OL2UV1NdFLyP770_BxDCbRbIfJxyghAQxVL9hpOXoMsXRu58-
> 41nFt2k-C-GdOu-Q44Q$  ). So provisional compliance with FIPS 140-2 level 1
> can be met for binaries linked to wrapper module when the system & kernel
> are fips enabled and the binary uses the required calling convention to
> autoconfigure or manually configure FIPS operation mode: We fork the
> pkgsrc repo and patch Makefiles for downstream builds' configure knobs
> where applicable to enable the calling of openssl's fips mode for those apps.
> I don't think this is worth correcting on the pkgsrc/NetBSD level for folks who
> need this niche requirement but I sent the message because the xz thread
> seemed related when dealing with a non-openssl-related base vs pkgsrc
> library issue...
> I bootstrapped on CentOS 7 using my modified bootstrap script (attached)
> with zlib-devel and openssl-devel preinstalled, so pkgsrc should be able to
> use native versions of both.  After installing libxml2, I verified that it's using
> pkgsrc zlib.  Packages that depend on openssl are not picking up the Yum
> installation, though, because it's one revision too old.  Yum's zlib meets the
> current requirements, though, so as a test I added it to PREFER_NATIVE and
> checked dependencies for libarchive before and after.  This confirmed that
> my mk.conf configuration is working as expected.
> 
> So I'm not sure why your installation was using pkgsrc zlib.  Did you have
> anything in PREFER_* besides PREFER_NATIVE=openssl?
> 
> Linux centosdev.ceas  bacon ~ 1072: (pkgsrc): yum list installed|grep zlib
> zlib.x86_64 1.2.7-18.el7                     @anaconda
> zlib-devel.x86_64 1.2.7-18.el7                     @base
> 
> Linux centosdev.ceas  bacon ~ 1073: (pkgsrc): yum list installed | grep openssl
> openssl.x86_64 1:1.0.2k-19.el7                  @anaconda
> openssl-devel.x86_64 1:1.0.2k-19.el7                  @base
> openssl-libs.x86_64 1:1.0.2k-19.el7                  @anaconda
> 
> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> 1074: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
> PREFER_PKGSRC=        yes PREFER_NATIVE=        openssl
> 
> [ Install pkgsrc libxml2 ]
> 
> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> 1075: (pkgsrc): ldd ~/Pkgsrc/pkg-test/lib/libxml2.so|grep libz
>      libz.so.1 => /home/bacon/Pkgsrc/pkg-test/lib/libz.so.1
> (0x00007fb48f8df000)
> 
> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> 1076: (pkgsrc): bmake show-depends
> bzip2>=1.0.3:../../archivers/bzip2
> xz>=5.0.0:../../archivers/xz
> zlib>=1.2.3:../../devel/zlib
> openssl>=1.1.1dnb2:../../security/openssl
> 
> [ Add zlib to PREFER_NATIVE ]
> 
> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> 1078: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
> PREFER_PKGSRC=        yes PREFER_NATIVE=        openssl zlib
> 
> Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
> 1079: (pkgsrc): bmake show-depends
> bzip2>=1.0.3:../../archivers/bzip2
> xz>=5.0.0:../../archivers/xz
> openssl>=1.1.1dnb2:../../security/openssl
> 

Libxml2 will pick up native libz fine when built first. The problem was www/nginx which pulled in pkgsrc libz while still linking to native openssl. After installing pkgsrc libz to satisfy nginx, libxml2 broke. Adding zlib to PREFER_NATIVE resolved the issue. I didn't have anything else in PREFER_NATIVE except for openssl