NetBSD 8.0, OpenSSL and Geode CPU

[manu%netbsd.org@localhost (Emmanuel Dreyfus)]

Hi 

On Geode GX processors, ldapsearch cannot perform a TLS bind using a GCM
cipher. 

Short version for the impatient, we have a workaround:
export OPENSSL_ia32cap=~0x800000

Now the long version. It will crash like this:

TLS trace: SSL_connect:SSLv3 read server certificate request A
TLS trace: SSL_connect:SSLv3 read server done A
TLS trace: SSL_connect:SSLv3 write client certificate A
TLS trace: SSL_connect:SSLv3 write client key exchange A
TLS trace: SSL_connect:SSLv3 write change cipher spec A
Illegal instruction (core dumped) 

gdb tells me it happens in OpenSSL's libcrypto:

Program terminated with signal SIGILL, Illegal instruction.
#0  0xbb96ad50 in gcm_ghash_4bit_mmx () from ./libcrypto.so.12
(gdb) bt
#0  0xbb96ad50 in gcm_ghash_4bit_mmx () from ./libcrypto.so.12
#1  0x00000000 in ?? ()
(gdb) x/1i $pc
=> 0xbb96ad50 <gcm_ghash_4bit_mmx+1104>:        pinsrw
$0x2,(%esi,%ebx,2),%mm2

I see in src/crypto/external/bsd/openssl/dist/crypto/modes/gcm128.c
that gcm_ghash_4bit_mmx() is an optimization and we can just use
gcm_ghash_4bit_x86() instead:

#  if   defined(GHASH_ASM_X86)  /* x86 only */
#   if  defined(OPENSSL_IA32_SSE2)
    if (OPENSSL_ia32cap_P[0] & (1 << 25)) { /* check SSE bit */
#   else
    if (OPENSSL_ia32cap_P[0] & (1 << 23)) { /* check MMX bit */    
#   endif
        ctx->gmult = gcm_gmult_4bit_mmx; 
        ctx->ghash = gcm_ghash_4bit_mmx; 
    } else {            
        ctx->gmult = gcm_gmult_4bit_x86;
        ctx->ghash = gcm_ghash_4bit_x86; 
    }
#  else                 
    ctx->gmult = gcm_gmult_4bit;
    ctx->ghash = gcm_ghash_4bit;
#  endif 

And OPENSSL_ia32cap_P is populated in
src/crypto/external/bsd/openssl/dist/crypto/cryptlib.c

    if ((env = getenv("OPENSSL_ia32cap"))) {
        int off = (env[0] == '~') ? 1 : 0;
#  if defined(_WIN32)
        if (!sscanf(env + off, "%I64i", &vec))
            vec = strtoul(env + off, NULL, 0);
#  else
        if (!sscanf(env + off, "%lli", (long long *)&vec))
            vec = strtoul(env + off, NULL, 0);
#  endif
        if (off)
            vec = OPENSSL_ia32_cpuid(OPENSSL_ia32cap_P) & ~vec;
        else if (env[0] == ':')
            vec = OPENSSL_ia32_cpuid(OPENSSL_ia32cap_P);

Documentation is here:
https://www.openssl.org/docs/man1.1.0/crypto/OPENSSL_ia32cap.html

MMX is bit 23, that is 0x800000

This disables MMX and lets ldapsearch bind with a GCM cipher:
export OPENSSL_ia32cap=~0x800000

Now why does it fail? OpenSSL considers pinsrw to be available with the
MMX feature, and Geode GX1 has it:

# cpuctl identify 0
cpu0: highest basic info 00000002
cpu0: highest extended info 80000005
cpu0: "Geode(TM) Integrated Processor by National Semi"
cpu0: National Semiconductor Geode GX1 (586-class)
cpu0: family 0x5 model 0x4 stepping 0 (id 0x540)
cpu0: features 0x808131<FPU,TSC,MSR,CX8,CMOV,MMX>
cpu0: ITLB 1 4KB entries 112-way
cpu0: Initial APIC ID 0

But looking up pinsrw on a search engine, all pages I find suggest is
should compe with SSE. Here is an example:
http://www.felixcloutier.com/x86/PINSRW.html

So is this a bug in OpenSSL? I would expect that if OPENSSL_IA32_SSE2 is
undefined, it would wrongly use MMX to decice when pinsrw is used, but I
see it is defined in
src/crypto/external/bsd/openssl/lib/libcrypto/arch/i386/Makefile

Is it somehow unused? And shouldn't OpenSSL abstain from using the MMX
version wih it finds MMX without SSE?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu%netbsd.org@localhost