Am Tue, 27 Feb 2018 09:18:05 -0600 schrieb Jason Bacon <bacon4000%gmail.com@localhost>: > dependent package > simply select one library (libblas, libatlas, libopenblas, > libopenblas-openmp, etc.) that's > known to work with it. And what about the possible conflicts/subtle unintended switching of actually active BLAS at runtime I elaborated on in my other post? I build the pkgsrc software partly so that users can build their software using the libraries provided by pkgsrc. Say, their own version of numpy. Or R packages. I will damn make sure that every package in my pkgsrc that uses some BLAS is using the _same_ BLAS to avoid any ambiguity when linking multiple libraries into one end user program. [parallel BLAS just one variant of BLAS] > I'm not sure it should be. As you pointed out before, some dependents > work better > with one library and others with a different one. Again, this is going Regarding implicit parallelizations, yes, some combinations can be problematic. That's why my favoured variant was to choose _one_ build of openblas to use for all BLAS needs in my install of pkgsrc, knowing that it will behave in our local setup. With OMP_NUM_THREADS=1, the openmp build works nicely with the otherwise problematic uses in R or numpy, just like the serial build. For official binary packages, I would choose a serial openblas as a safe choice after all (for platforms that are supported). Factory default might stay netlib BLAS, as it's slow (depending on how good the compiler is), but stable. But: The BLAS API as such is a standard. It is one set of symbols provided by multiple implementations of the library. As long as we are talking about serial implementations, every package should work with every BLAS. The parallel ones, even if they work, may not be what you want in all setups. Just observe how openblas actively avoids creating more parallelism if you call it inside an OpenMP parallel region. As a packager, you cannot predict the needs of the sites. > to soak up a lot > of man hours that we could be using to create and maintain other > important packages. Yes, properly handling several parallel installs of BLAS packages would be a headache. My point is that installing multiple BLAS packages next to each other and _not_ managing them in a consistent way would also be a huge headache, with fun PRs resulting. > I favor an approach the produces reliable dependent packages first and > foremost. Hence I suggest resisting the urge to install multiple openblas libraries at the same time. Chose one default for the platform (serial), offer choice to admin to select a different one at openblas build time. I'd even go so far as to avoid installing netlib BLAS alongside openblas. > Improving performance on a per-package basis would be my next priority, > followed > by interchangeability. To be honest: I'm tempted to have pkgsrc use generic/external BLAS only and handle the interchangeable builds of openblas and friends outside pkgsrc. If at all. I am also looking at EasyBuild, which is a framework for packaging software that was created to partly do what I do with pkgsrc, in the HPC context. They group BLAS into the toolchain domain, along with the compiler and MPI library. Maybe it is worth considering this functionality as a low-level one, basic as the choice of a compiler (and at the same time, something you can switch at runtime, at the highest level, but only because it has a stable ABI). I'm willing to give up the run-time switchery for simplicity. But only if that indeed results in getting rid of problems. Just had a team discussion again … after attempts to explain the situation and the various options about interchanging BLAS or not, we settled on the plan to have our install of pkgsrc feature exactly _one_ BLAS variant. If people want to shoot around in the general foot area, they can play games with other builds of (Open)BLAS and LD_LIBRARY_PATH or perhaps better LD_PRELOAD. This will be fine if you know what you're doing with your application. It would be nice if our kind of installation would be a possible option of a normal pkgsrc tree. Otherwise I'll carry it as patches like I'm carrying lots of patches already (with varying success of getting them upstreamed). > The problem I see is that the various BLAS implementations are not > freely interchangeable > in reality, even though they are meant to be. You do agree, though, that serial implementations are (should) be interchangeable? > Leaving it to the > end-user to switch between > them at run-time is going to cause problems and generate a lot of PRs. But trying to decide for all users at build-time will not always be the correct guess. Even if they are using the same software, some might want to employ parallel BLAS, some might require a serial one. In the case of stuff like numpy or R, this depends on the R script you run, not on the R binary you use. And you have the possible interference of combining packages (libraries) that (try to) employ different BLAS. You will have unhappy users either way. Avoiding problems means, for me, to avoid a possible mix-up of BLAS implementations in the tree, in a running application. There might be cases where openblas-pthread/openmp might not work with a certain package, but apart from the serial/parallel mess: Do you know any application that will _not_ work with -lopenblas instead of -lblas -llapack? > I think things will go much more smoothly if the choice is left to the > packager rather than the end-user. The packager does not have the information needed for the decision. I don't see anything sane apart from settling on serial openblas for supported platforms as a default, leaving the admin/user to decide to override this for a parallel one in a local build, knowing what the actual users need. > This way we don't have to test (blas packages) * (dependent packages) > different configurations and retest > every dependent package every time *any* blas package is updated. Every platform has one BLAS. Every package shall use that by default and that is what is tested in bulk builds. What kind of tests will you do? Any variant of openblas will build/link just fine with any dependent package. The possible issues arise at runtime in specific ways of using the packages. Do you want to test that? > of prioritizing limited manpower. I have a long list of new scientific > packages in the queue that I think would > be more valuable to the pkgsrc community than interchangeable blas binaries. OK, then. We're halfway agreeing already. You just need to let go of wanting to install multiple BLAS libraries next to each other, too;-) Going back to the details of what to install from openblas. We can set NO_CBLAS=1 to pull cblas.h and libcblas from the cblas package instead. We could contemplate setting NO_LAPACK=1, too, relying on the netlib LAPACK only, but we miss some optimizations, then. So, I'd rather include it. It boils down to deciding which variants of BLAS support are supported as a combination. I imagine creating those: mk/lapack.mk mk/blas.mk mk/lapacke.mk mk/cblas.mk In the first incarnation, they all use a single variable to decide on netlib/atlas/openblas/serial/parallel. Depending packages use variables exported by these fragments (BLAS_LIBS, LAPACK_LIBS, etc.) and insert those in build configuration as needed. The cblas and lapacke ones would just point to the netlib implementations, which then depend on mk/blas and mk/lapack. I really, really do not want to hack each package for building with a certain BLAS when multiple libraries with differing names are installed. That would eat my time and nerves. A second incarnation could try to play games with combining things like netlib LAPACK and openblas BLAS. Or get cblas from openblas, too. We might never reach this second stage given developer time constraints, but the path would be clear and there is no breakage of existing infrastructure needed. A third incarnation could introduce interchangeable BLAS and friends and have the above fragments default BLAS_LIBS=-lblas as there are wrapper/symlink libraries with that name available. No changes needed to dependent packages at all. Again, things would still be OK if we never reach this stage. Also, I think pkgsrc maintainers should only be concerned with testing and supporting the default choice for the platform. I for sure won't build and test atlas just for fun. There might be subtleties once LAPACK version increases. But as long as you have one defined default BLAS+LAPACK for a platform, this gets tested. It would only be an issue for run-time switching. > I think the important thing is to make all implementations available, > without install conflicts. I'm agnostic > about how this is done. There I disagree. Well, it might be OK to install both atlas and openblas without conflicts (neither of them installing a libblas.so), but I strongly oppose packages in pkgsrc sometimes using one or the other. This is the madness that leads to subtle bugs. As the libraries heavily overlap in the symbols they offer, we really should ensure they never get loaded together in package builds. If the end user decides to use atlas in a build of their own software where pkgsrc used openblas, then it's on them to sort that out. And they have the chance to see where it went wrong. But if libraries from pkgsrc already pull in differing libraries with overlapping symbols, that's just bad form. > As long as I can make a dependent package use any of the available blas > implementations and I can install > them all on the same cluster, I'm happy. Of course you can, by a) implementing run-time override of the dynamic libs or b) installing multiple pkgsrc trees with differing choices You can automate b) of course. You can achieve a) by LD_PRELOADing one of your other installed BLAS choices in the tree. You can even load libatlas.so to override libopenblas.so, except for the symbols specific to openblas (openblas_set_num_threads()). I even suspect that this would work where openblas_set_num_threads() is called. It just wouldn't set the threads for the other implementation. Or it would, by accident. I want to stress this: Do you really _not_ see multiple different BLAS libs being pulled in by a dependency chain as a problem? Alrighty then, Thomas -- Dr. Thomas Orgis Universität Hamburg RRZ / Basis-Infrastruktur / HPC Schlüterstr. 70 20146 Hamburg Tel.: 040/42838 8826 Fax: 040/428 38 6270
Attachment:
smime.p7s
Description: S/MIME cryptographic signature