Hi,
I had the thought that pkgsrc would be a nice way to provide a usable
set of userspace software on aging HPC installations (where the base
GNU/Linux system cannot be upgraded without endangering the operation).
Well, it might be useful also on a fresh installation, since in HPC, we
have some funny ideas about software management. Two main points:
1. If there is one version of a certain software (especially numerical
library) at one time, it should stay there and be usable as long as
possible (lifetime of the system). This comes from the basic
requirement to be able to reproduce scientific computations. You would
also like to be able to reproduce your binaries, which means the exact
same toolchain you used before should be still there.
2. We like to (or have to) use several differing compilers. There are
compilers that work better for different software. When
PGI/Intel/Open64/SStudio/whatever builds 10% faster binaries for a
specific code, it will be used. The presence of concurrent compilers
(and versions of these, see point 1) can cause major headache
especially when C++ or Fortran 95 code is in the game, with associated
ABI-incompatible libraries.
Now, pkgsrc seems to offer the option to install a consistent tree of
userspace software into a prefix and keep it there. When the next
version arrives, one can install a fresh pkgsrc tree into another
location. It also offers some libraries that user programs would like
to use, like devel/netcdf or parallel/openmpi.
When we want the option of using pkgsrc-provided libraries in
scientific binaries, or just want adequate performance out of math/R,
for example, we want to build pkgsrc using a recent and/or 3rd party
compiler.
The trouble starts when trying to make pkgsrc use one of our
separately installed compilers. A recent version of gcc might be
included in pkgsrc, but perhaps we want to use the Intel compiler,
which seems to be supported by pkgsrc.
So ... I tried setting up use of a build of GCC we have, including C++
and Fortan, installed under /sw/compilers/gcc-4.8.2 .
This works as long as I don't have the funny idea that the runtime
libraries for C++ and Fortran should be found by pkgsrc programs
without manupulating LD_LIBRARY_PATH to that effect. We would like to
avoid polluting the user's environment and influence the users' programs
just because one wants to _run_ a program from pkgsrc.
I tried setting some LDFLAGS with RPATH in mk.conf, I added variables
needed to let pkgsrc pass these settings along instead of filtering
out ... in the end, without real success. I had some binaries built
with proper library path /sw/compilers/gcc-3.8.2/lib burned in, so that
they find libgfortran.so, at some point. But I never had a consistently
working pkgsrc process that didn't product funky issues at some place.
One symptom from trying to force library paths into pkgsrc are mangled
compiler/linker flags that are just ... weird. Shortened example:
gcc -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN \
-DHAVE_DLFCN_H -O2 \
-Wno-error \
... \
/sw/compilers/gcc-4.8.2/lib64 /sw/compilers/gcc-4.8.2/lib \
... \
libcrypto.a -Wl,-R -Wl,-R -Wl,-R -Wl,-R \
... \
-Wl,-R/sw/with-compiler/gcc-4.8.2/pkgsrc-2014Q2/lib -ldl
This I got in the openssl build while having these settings in mk.conf:
PKGSRC_COMPILER= gcc
CFLAGS+= -march=native -O2
LDFLAGS+= -Wl,-R /sw/compilers/gcc-4.8.2/lib64 -Wl,-R
/sw/compilers/gcc-4.8.2/lib
BUILDLINK_PASSTHRU_DIRS+= /sw/compilers/gcc-4.8.2/lib64
/sw/compilers/gcc-4.8.2/lib
BUILDLINK_PASSTHRU_RPATHDIRS+= /sw/compilers/gcc-4.8.2/lib64
/sw/compilers/gcc-4.8.2/lib
I also observe strange library flags in the installed
libtool-fortran ... starting with "-l -l -L-L", which obviously is
rubbish. I removed these already, though.
I won't tell all detail about what I tried and what didn't work. I did
discover these settings:
GCCBASE=/sw/compilers/gcc-4.8.2
USE_NATIVE_GCC=yes
.. hoping that pkgsrc, since it insists on mangling compiler/linker
flags itself, would figure out that it should set rpath to include the
library dirs in that prefix. The assumption would have to be that all
libs under $GCCBASE do matter (different from /usr/lib). But this
doesn't work. While it finds the gcc binary, the Fortran runtime stays
in lala-land. The openmpi build tries this:
configure:28741: gfortran -o conftest -O
-L/sw/with-compiler/gcc-4.8.2/pkgsrc-2014Q2/lib
-Wl,-R/sw/with-compiler/gcc-4.8.2/pkgsrc-2014Q2/lib conftest.f >&5
configure:28741: $? = 0
configure:28741: ./conftest
./conftest: error while loading shared libraries: libgfortran.so.3: cannot open
shared object file: No such file or directory
It may call the correct gfortran binary, but nowhere is it told where
to pull libgfortran.so.3 from.
Now, you could argue that GCC is just broken when it doesn't hardcode
its library path itself, but this is moot: This is how GCC works. We
need to deal with it.
Now I do wonder:
1. Is there a working way to have pkgsrc use an externally provided
compiler and include its runtime library path in the magic it does with
buildlinks3™? I just wondered if symlinking the libs into pkgsrc's
library directory would work. Perhaps. Would that be the proper solution?
2. Or is there some hidden setting that I can enforce pkgsrc to include
-Wl,-R /sw/compilers/gcc-4.8.2/lib64 in LDFLAGS and _not_ _mess_ _with_
_that_, as this is all that is missing. It boils down to one stupid
directory that should be baked into the libraries in addition to
$PKGSRC_PREFIX/lib. I admit that I'm rather annoyed that pkgsrc insists
on knowing better ... and that the documentation doesn't tell me that
right away.
Or, am I completely misguided? If pkgsrc doesn't want to play with me,
I may just resort to using it with its own gcc and ditch the idea to
ease builds of scientific software contained in pkgsrc with consistent
toolchains. It would reduce the utility I could get from pkgsrc and
that would be a shame, wouldn't it? Because, if it fully works for our
setup, I'd be also tempted to add more software we use to pkgsrc ...
Alrighty then,
Thomas
--
Dr. Thomas Orgis
Universität Hamburg
RRZ / Zentrale Dienste / HPC
Schlüterstr. 70
20146 Hamburg
Tel.: 040/42838 8826
Fax: 040/428 38 6270
Attachment:
smime.p7s
Description: S/MIME cryptographic signature