pkgsrc-Users archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles that include devel/zlib/buildlink3.mk

On 2020-08-15 10:56, Lai, Peter PW wrote:

-----Original Message-----
From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
Sent: Friday, August 14, 2020 5:23 PM
To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>; pkgsrc-users%netbsd.org@localhost
Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash for Makefiles
that include devel/zlib/buildlink3.mk

On 2020-08-10 12:05, Lai, Peter PW wrote:

-----Original Message-----
From: Jason Bacon [mailto:outpaddling%yahoo.com@localhost]
Sent: Monday, August 10, 2020 10:00 AM
To: Lai, Peter PW <peter.lai2%prattwhitney.com@localhost>;
pkgsrc-users%netbsd.org@localhost
Subject: [External] Re: RHEL/Centos 7 builtin vs pkgsrc zlib clash
for Makefiles that include devel/zlib/buildlink3.mk

On 2020-08-10 05:50, Lai, Peter PW wrote:

Environment:

* RHEL 7.7 (Centos 7 for those playing at home I guess) in Azure.
* Toolchain: Software collection (SCL) devtoolset-9

* Yum installs native zlib-devel as a dependency for openssl-devel

* openssl-devel was installed in order to support PREFER_NATIVE=

openssl

Observations:

In this environment because zlib-devel is installed, bootstrap by
default

picks up native zlib (/usr/include/zlib.h). When devel/libxml2 is
then built, it is built against native zlib (libz.so.1 => /lib64/libz.so.1).

Next, some packages that also include
"../../devel/zlib/buildlink3.mk" such

nginx is built, it pulls in devel/zlib. This is weird because RHEL 7
includes zlib
1.2.7 which satisfies zlib>=1.2.3 in buildlink3.

What happens after is that libxml2 and downstream libxml2
dependencies

(e.g. databases/postgresql) break because they see the pkgsrc zlib first:

/mnt/datadisk01/pkg/bin/postgres: /mnt/datadisk01/pkg/lib/libz.so.1:
no version information available (required by
/mnt/datadisk01/pkg/lib/libxml2.so.2)

# ldd /mnt/datadisk01/pkg/lib/libxml2.so.2
/mnt/datadisk01/pkg/lib/libxml2.so.2:
/mnt/datadisk01/pkg/lib/libz.so.1: no version information available
(required by /mnt/datadisk01/pkg/lib/libxml2.so.2)
/mnt/datadisk01/pkg/lib/libxml2.so.2:
/mnt/datadisk01/pkg/lib/libz.so.1: no version information available
(required by /mnt/datadisk01/pkg/lib/libxml2.so.2)

I worked around this by adding zlib to PREFER_NATIVE when
re-building

nginx, but is there a better way to resolve this? This looks somewhat
related to the continuing "pkg_install fails when archivers/xz is
installed" thread because RPATH manipulation could be involved, but I
am opening a separate thread in case it is not.
This does look like the same issue that we've been discussing
regarding xz and it's a great example of the kind of problems that
are avoided by using PREFER_PKGSRC=yes (and/or PREFER_NATIVE=no).

What's your motive for using native openssl instead of pkgsrc?  The
only issue I've seen with the latter is the need to manually install
security/mozilla-rootcerts* or equivalent to make curl, etc. work with

https.

If there's some other disadvantage to pkgsrc openssl, maybe we should
try to correct it.

The RHEL version of OpenSSL is backported/patched 1.0.2k with the FIPS

module compiled (and validated through vendor affirmed basis as per
https://urldefense.com/v3/__https://www.openssl.org/docs/fips.html__;!!
MvWE!V8OL2UV1NdFLyP770_BxDCbRbIfJxyghAQxVL9hpOXoMsXRu58-
41nFt2k-C-GdOu-Q44Q$  ). So provisional compliance with FIPS 140-2 level 1
can be met for binaries linked to wrapper module when the system & kernel
are fips enabled and the binary uses the required calling convention to
autoconfigure or manually configure FIPS operation mode: We fork the
pkgsrc repo and patch Makefiles for downstream builds' configure knobs
where applicable to enable the calling of openssl's fips mode for those apps.
I don't think this is worth correcting on the pkgsrc/NetBSD level for folks who
need this niche requirement but I sent the message because the xz thread
seemed related when dealing with a non-openssl-related base vs pkgsrc
library issue...
I bootstrapped on CentOS 7 using my modified bootstrap script (attached)
with zlib-devel and openssl-devel preinstalled, so pkgsrc should be able to
use native versions of both.  After installing libxml2, I verified that it's using
pkgsrc zlib.  Packages that depend on openssl are not picking up the Yum
installation, though, because it's one revision too old.  Yum's zlib meets the
current requirements, though, so as a test I added it to PREFER_NATIVE and
checked dependencies for libarchive before and after.  This confirmed that
my mk.conf configuration is working as expected.

So I'm not sure why your installation was using pkgsrc zlib.  Did you have
anything in PREFER_* besides PREFER_NATIVE=openssl?

Linux centosdev.ceas  bacon ~ 1072: (pkgsrc): yum list installed|grep zlib
zlib.x86_64 1.2.7-18.el7                     @anaconda
zlib-devel.x86_64 1.2.7-18.el7                     @base

Linux centosdev.ceas  bacon ~ 1073: (pkgsrc): yum list installed | grep openssl
openssl.x86_64 1:1.0.2k-19.el7                  @anaconda
openssl-devel.x86_64 1:1.0.2k-19.el7                  @base
openssl-libs.x86_64 1:1.0.2k-19.el7                  @anaconda

Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1074: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
PREFER_PKGSRC=        yes PREFER_NATIVE=        openssl

[ Install pkgsrc libxml2 ]

Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1075: (pkgsrc): ldd ~/Pkgsrc/pkg-test/lib/libxml2.so|grep libz
      libz.so.1 => /home/bacon/Pkgsrc/pkg-test/lib/libz.so.1
(0x00007fb48f8df000)

Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1076: (pkgsrc): bmake show-depends
bzip2>=1.0.3:../../archivers/bzip2
xz>=5.0.0:../../archivers/xz
zlib>=1.2.3:../../devel/zlib
openssl>=1.1.1dnb2:../../security/openssl

[ Add zlib to PREFER_NATIVE ]

Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1078: (pkgsrc): grep PREFER ~/Pkgsrc/pkg-test/etc/mk.conf
PREFER_PKGSRC=        yes PREFER_NATIVE=        openssl zlib

Linux centosdev.ceas  bacon ~/Pkgsrc/pkgsrc-test/archivers/libarchive
1079: (pkgsrc): bmake show-depends
bzip2>=1.0.3:../../archivers/bzip2
xz>=5.0.0:../../archivers/xz
openssl>=1.1.1dnb2:../../security/openssl


Libxml2 will pick up native libz fine when built first. The problem was www/nginx which pulled in pkgsrc libz while still linking to native openssl. After installing pkgsrc libz to satisfy nginx, libxml2 broke. Adding zlib to PREFER_NATIVE resolved the issue. I didn't have anything else in PREFER_NATIVE except for openssl
So in the absence of a PREFER_PKGSRC setting and PREFER_NATIVE set to a list of specific packages, dependencies not in the PREFER_NATIVE list will be linked preferentially from pkgsrc over native if both are installed, otherwise to whichever is available. This by itself seems fairly reasonable to me, with some obvious risks especially on Linux.  The problem here is that it doesn't account for pre-installed packages linked against a native dependency before the pkgsrc equivalent dependency was installed.
Bookstrapping with --prefer-pkgsrc (which will soon be the default for Linux) should avoid this issue by installing the pkgsrc dep regardless of whether there's a native equivalent available.
I don't see a simple way to address this issue without setting PREFER_PKGSRC during bootstrap.  Trying to avoid these issues using RPATH/RUNPATH or equivalent is looking a bit messy, if it's even feasible.
I still see a slight potential for problems if someone explicitly installs a pkgsrc package listed in PREFER_NATIVE, or if a bug causes it to be installed as a dep, but I'm not sure that's worth trying to prevent. 

    JB
Home | Main Index | Thread Index | Old Index