dlopen() and libpthread

[Emmanuel Dreyfus <manu%netbsd.org@localhost>]

Hi

After upgrading from NetBSD-5 to NetBSD-6, OpenSSH PKCS#11 (for
cryptographic tokens) support broke.

Here is how it is supposed to work:
  $ eval $( /usr/pkg/bin/ssh-agent )
  Agent pid 15183
  $ ssh-add -s /usr/pkg/lib/opensc-pkcs11.so            
  Enter passphrase for PKCS#11: 
  Card added: /usr/pkg/lib/opensc-pkcs11.so

On NetBSD-6 ssh-add -s always fails. Since ssh-add just remote control 
ssh-agent, this is in fact a ssh-agent failure. We can see it in 
/var/log/authlog:
  ssh-pkcs11-helper[15047]: error: dlopen /usr/pkg/lib/opensc-pkcs11.so failed:
  Cannot dlopen non-loadable /usr/lib/libpthread.so.

ssh-agent -d just gives unhelpful messages which I only reproduce there so
that web crawler index it and people lookinf for an answer get a chance:
  debug1: type 20 
  debug1: XXX shrink: 3 < 4

Exploring dynamic linker code, it is obvious that this conditions trigger
the error, in src/libexec/ld.elf_so/load.c
                if (flags & _RTLD_DLOPEN) {
                        if (obj->z_noopen || (flags & _RTLD_NOLOAD)) {

obj->z_noopen is a flag from the ELF dynamic header which means the 
shared object shall not be loaded by dlopen(3). We can explore it
using readelf(1):

  $ readelf -d /usr/lib/libpthread.so.1|grep FLAG
   0x6ffffffb (FLAGS_1)                    Flags: NOOPEN
  $ uname -sr
  NetBSD 6.0
  
  $ readelf -d /usr/lib/libpthread.so.0|grep FLAG 
  $ uname -sr
  NetBSD 5.0.2

This is why it stopped working on NetBSD-6.0 upgrade: NetBSD-6.0 libpthread
cannot be loaded by dlopen(3) while  that was possible on NetBSD-5.x. 

We have a workaround, which is to tell the dynamic linker that the shared
object shall be loaded on startup time, so that it does not need to load
it using dlopen(3) later. This fixes the problem on NetBSD-6.0:
  $ export LD_PRELOAD=/usr/lib/libpthread.so.1
  $ eval $( /usr/pkg/bin/ssh-agent )
  Agent pid 15183
  $ ssh-add -s /usr/pkg/lib/opensc-pkcs11.so            
  Enter passphrase for PKCS#11: 
  Card added: /usr/pkg/lib/opensc-pkcs11.so

And my question is: why should libpthread be forbidden for dlopen(3)?

-- 
Emmanuel Dreyfus
manu%netbsd.org@localhost