www/ap2-wsgi (apache mod_wsgi): unable to find pkgsrc python2.7 on system with native python2.7

[Matthias Ferdinand <mf+ml.pkgsrc-users%netzwerkagentursaarland.de@localhost>]

OS: Ubuntu 14.04 x86_64

apache / python2.7 / py27-moin / ap2-wsgi from pkgsrc (at /opt/pkgsrc)

I am in the process of migrating a moinmoin wiki instance from an Ubuntu
8.04 system (native python: 2.5.2) to Ubuntu 14.04 (native python:
2.7.5). Because of the similar versions in native system and pkgsrc,
mod_wsgi looks in the wrong place for some module imports.

On the 14.04 system (with identical pkgsrc packages and apache
configuration as the Ubuntu 8.04 system), apache would never reach the
point that it would accept requests, instead it would continously add
python tracebacks to the error log:

    ImportError: No module named _sysconfigdata_nd
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site.py", line 563, in <module>
        main()
      File "/usr/lib/python2.7/site.py", line 545, in main
        known_paths = addusersitepackages(known_paths)
      File "/usr/lib/python2.7/site.py", line 272, in addusersitepackages
        user_site = getusersitepackages()
      File "/usr/lib/python2.7/site.py", line 247, in getusersitepackages
        user_base = getuserbase() # this will also set USER_BASE
      File "/usr/lib/python2.7/site.py", line 237, in getuserbase
        USER_BASE = get_config_var('userbase')
      File "/usr/lib/python2.7/sysconfig.py", line 578, in get_config_var
        return get_config_vars().get(name)
      File "/usr/lib/python2.7/sysconfig.py", line 524, in get_config_vars
        _init_posix(_CONFIG_VARS)
      File "/usr/lib/python2.7/sysconfig.py", line 408, in _init_posix
        from _sysconfigdata import build_time_vars
      File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
        from _sysconfigdata_nd import *

As you can see, mod_wsgi is looking in the wrong place for
_sysconfigdata.py (/usr/lib/python2.7/ instead of
/opt/pkgsrc/lib/python2.7/).

Setting "python-path" in the "WSGIDaemonProcess" doesn't help:

            WSGIDaemonProcess moinmoind user=www-data group=www-data processes=1 threads=10 maximum-requests=1000 umask=0007 python-eggs=/opt/pkgsrc/var/cache/wsgi-python-eggs python-path=/opt/pkgsrc/lib/python2.7:/opt/pkgsrc/lib/python2.7/plat-linux2:/opt/pkgsrc/lib/python2.7/lib-tk:/opt/pkgsrc/lib/python2.7/lib-old:/opt/pkgsrc/lib/python2.7/lib-dynload:/opt/pkgsrc/lib/python2.7/site-packages 


strace excerpt:

    stat("/opt/pkgsrc/bin/python", 0x7fffa5d47850) = -1 ENOENT (No such file or directory)
    stat("/opt/pkgsrc/sbin/python", 0x7fffa5d47850) = -1 ENOENT (No such file or directory)
    stat("/usr/local/sbin/python", 0x7fffa5d47850) = -1 ENOENT (No such file or directory)
    stat("/usr/local/bin/python", 0x7fffa5d47850) = -1 ENOENT (No such file or directory)
    stat("/usr/sbin/python", 0x7fffa5d47850) = -1 ENOENT (No such file or directory)
    stat("/usr/bin/python", {st_mode=S_IFREG|0755, st_size=165, ...}) = 0
    readlink("/usr/bin/python", "python2.7", 4096) = 9
    readlink("/usr/bin/python2.7", 0x7fffa5d49900, 4096) = -1 EINVAL (Invalid argument)
    stat("/usr/bin/Modules/Setup", 0x7fffa5d477a0) = -1 ENOENT (No such file or directory)
    stat("/usr/bin/lib/python2.7/os.py", 0x7fffa5d47790) = -1 ENOENT (No such file or directory)
    stat("/usr/bin/lib/python2.7/os.pyc", 0x7fffa5d47790) = -1 ENOENT (No such file or directory)
    stat("/usr/lib/python2.7/os.py", {st_mode=S_IFREG|0644, st_size=25769, ...}) = 0

Note that it never _executes_ /usr/bin/python.
Apparently it looks for a "python" executable and derives module
locations from its path.

Workarounds:

  a) remove /usr/bin/python
     - bad, because /usr/bin/python is used by some Ubuntu base system
       software

  b) add symlink /opt/pkgsrc/bin/python => /opt/pkgsrc/bin/python2.7
     - workable, but not elegant, you would have to do that
       manually on every mod_wsgi installation

It is strange that mod_wsgi searches for a "python" executable, even
though it was provided with a full path to it at compile time
(CONFIGURE_ARGS+= --with-python=${PYTHONBIN:Q} ). And then there is also
the explicit "python-path" setting in the configuration, obviating the
need for autodetection.

Any ideas how to get mod_wsgi on the right track?

Or if this is just the way (embedded) python works, perhaps the pkgsrc
package should offer a python=>python2.7 symlink as in the Ubuntu base
system?

Regards
Matthias