lib/54055: Incorrect error reporting for SSL_CTX_set_cipher_list with OpenSSL 1.1

[Thomas Klausner <wiz%NetBSD.org@localhost>]

>Number:         54055
>Category:       lib
>Synopsis:       Incorrect error reporting for SSL_CTX_set_cipher_list with OpenSSL 1.1
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Mar 11 15:55:00 +0000 2019
>Originator:     Thomas Klausner
>Release:        NetBSD 8.99.35
>Organization:
Curiosity is the very basis of education and if you tell me that 
curiosity killed the cat, I say only that the cat died nobly.
- Arnold Edinborough
>Environment:
Architecture: x86_64
Machine: amd64
>Description:
When debugging a mercurial test problem, we found out that OpenSSL
1.1 on NetBSD-current behaves different that OpenSSL 1.0 on NetBSD
7 (and gentoo), but also than OpenSSL 1.1 on Debian SID. So this
seems to be a problem specific to the 1.1 version of OpenSSL on
NetBSD.

The problem appears with both clang and gcc.

Mercurial bug report for completeness:
https://bz.mercurial-scm.org/show_bug.cgi?id=6030

>How-To-Repeat:
Compile the following program and run it, e.g. with
"gcc -Wall -o ssltest ssltest.c -lcrypto -lssl"

#include <openssl/ssl.h>

int
main(int argc, char *argv[])
{
    SSL_CTX *ctx = NULL;

    /* import ssl */
    OPENSSL_add_all_algorithms_noconf();
    SSL_load_error_strings();
    SSL_library_init();

    /* ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1)  # or ssl.PROTOCOL_SSLv23 ? */
    ctx = SSL_CTX_new(TLSv1_method());
    if (ctx == NULL) {
        fprintf(stderr, "error creating context\n");
        return 1;
    }
    /* ctx.set_ciphers("HIGH")  # works */
    if (SSL_CTX_set_cipher_list(ctx, "HIGH") == 0) {
        fprintf(stderr, "HIGH: No cipher can be selected.\n");
        return 1;
    }

    /* ctx.set_ciphers("invalid")  # SSLError('No cipher can be selected.') */
    if (SSL_CTX_set_cipher_list(ctx, "invalid") == 0) {
        fprintf(stderr, "invalid: No cipher can be selected.\n");
        return 1;
    }

    return 0;
}

On NetBSD 7 (and gentoo) with OpenSSL 1.0, and on Debian SID with
OpenSSL 1.1 this correctly reports:

invalid: No cipher can be selected.

On NetBSD 8.99.35 with OpenSSL 1.1 it completes without output and
returns 0.

>Fix:
Yes, please.

>Unformatted: