ssl code going astray

[Thomas Klausner <wiz%NetBSD.org@localhost>]

Hi!

I have a problem with wip/Sigil where I don't know whose fault it is
and how to fix it, or at least work around it.

Sigil uses an embedded python to run python scripts, including one
that checks if an updated version is available - just an https request
and parsing the output.

The https request goes through python38's ssl module, which uses
openssl[1]. So far the theory, and this seems to work e.g. on Arch
Linux.

On my NetBSD 9.99.81/amd64 however, this dumps core, with a weird
backtrace[2]: In the middle of the openssl part of the stack (frames
16-5), functions from nss are called instead of the ones of the same
name from openssl (frames 4-0).

Sigil is linked against both qt5-qtbase (using openssl) and
qt5-qtwebengine (using nss).

I don't understand why at this point in the code the functions from
nss are preferred to the ones from openssl, or how to change that.

Can anyone explain the behaviour?

What is the solution?

Or at least, what is a workaround?

All explanations very welcome.

More details in the upstream bug report[3].

Thanks,
 Thomas


[1]
# ldd /usr/pkg/lib/python3.8/lib-dynload/_ssl.so
/usr/pkg/lib/python3.8/lib-dynload/_ssl.so:
        -lssl.14 => /usr/lib/libssl.so.14
        -lcrypto.14 => /usr/lib/libcrypto.so.14
        -lcrypt.1 => /lib/libcrypt.so.1
        -lc.12 => /usr/lib/libc.so.12
        -lpthread.1 => /usr/lib/libpthread.so.1

[2]
#0  0x0000781b56a16d27 in SHA512_Update (ctx=0x781b89d3d2cb <init384>, input=0x781b9c2969f0 "", inputLen=48) at sha512.c:1338
#1  0x0000781b56a173d1 in SHA384_Update (ctx=<optimized out>, input=<optimized out>, inputLen=<optimized out>) at sha512.c:1539
#2  0x0000781b972494b2 in SHA384_Update (cx=0x781b89d3d2cb <init384>, input=0x781b9c2969f0 "", inputLen=48) at loader.c:1110
#3  0x0000781b56a181c3 in HMAC_Update (cx=<optimized out>, data=<optimized out>, data_len=<optimized out>) at alghmac.c:165
#4  0x0000781b9724aa70 in HMAC_Update (cx=0x781ba9ffae20, data=0x781b9c2969f0 "", data_len=48) at loader.c:1524
#5  0x0000781b89d31de9 in HMAC (evp_md=0x781b8a0a57e0, key=0x781b89e17503, key_len=0, d=0x781b9c2969f0 "", n=48, md=md@entry=0x781b66f30d3c "", md_len=md_len@entry=0x7f7ffff4b29c)
    at /usr/src/crypto/external/bsd/openssl/dist/crypto/hmac/hmac.c:231
#6  0x0000781b89d31139 in HKDF_Extract (prk_len=0x7f7ffff4b378, prk=0x781b66f30d3c "", key_len=<optimized out>, key=<optimized out>, salt_len=<optimized out>, salt=<optimized out>, evp_md=<optimized out>)
    at /usr/src/crypto/external/bsd/openssl/dist/crypto/kdf/hkdf.c:283
#7  pkey_hkdf_derive (ctx=<optimized out>, key=0x781b66f30d3c "", keylen=0x7f7ffff4b378) at /usr/src/crypto/external/bsd/openssl/dist/crypto/kdf/hkdf.c:215
#8  0x0000781b8a246529 in tls13_generate_secret (s=s@entry=0x781b66f30c00, md=0x781b8a0a57e0, prevsecret=0x781b8a266300 "", prevsecret@entry=0x0, insecret=0x781b8a266300 "", insecret@entry=0x0, insecretlen=48, insecretlen@entry=0, 
    outsecret=outsecret@entry=0x781b66f30d3c "") at /usr/src/crypto/external/bsd/openssl/dist/ssl/tls13_enc.c:241
#9  0x0000781b8a25e67c in ssl_derive (s=s@entry=0x781b66f30c00, privkey=privkey@entry=0x781b9c2edb10, pubkey=pubkey@entry=0x781b9c2edac0, gensecret=gensecret@entry=1)
    at /usr/src/crypto/external/bsd/openssl/dist/ssl/s3_lib.c:4821
#10 0x0000781b8a24381f in tls_parse_stoc_key_share (s=0x781b66f30c00, pkt=<optimized out>, context=<optimized out>, x=<optimized out>, chainidx=<optimized out>)
    at /usr/src/crypto/external/bsd/openssl/dist/ssl/statem/extensions_clnt.c:1885
#11 0x0000781b8a245428 in tls_parse_all_extensions (s=s@entry=0x781b66f30c00, context=context@entry=512, exts=0x781b9c2f1900, x=x@entry=0x0, chainidx=chainidx@entry=0, fin=fin@entry=1)
    at /usr/src/crypto/external/bsd/openssl/dist/ssl/statem/extensions.c:750
#12 0x0000781b8a23728d in tls_process_server_hello (s=0x781b66f30c00, pkt=<optimized out>) at /usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1692
#13 0x0000781b8a239d65 in ossl_statem_client_process_message (s=<optimized out>, pkt=<optimized out>) at /usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_clnt.c:1033
#14 0x0000781b8a23a751 in read_state_machine (s=0x781b66f30c00) at /usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem.c:636
#15 state_machine (s=0x781b66f30c00, server=0) at /usr/src/crypto/external/bsd/openssl/dist/ssl/statem/statem.c:434
#16 0x0000781b8a256bdf in SSL_do_handshake (s=0x781b66f30c00) at /usr/src/crypto/external/bsd/openssl/dist/ssl/ssl_lib.c:3672
#17 0x0000781b5741922d in _ssl__SSLSocket_do_handshake_impl (self=0x781b9c3a7ac0) at /scratch/lang/python38/work/Python-3.8.8/Modules/_ssl.c:1087
#18 _ssl__SSLSocket_do_handshake (self=0x781b9c3a7ac0, _unused_ignored=<optimized out>) at /scratch/lang/python38/work/Python-3.8.8/Modules/clinic/_ssl.c.h:19

[3] https://github.com/Sigil-Ebook/Sigil/issues/604