standards/52819: a64l should sign-extend its result when long has more than 32 bit

[daniel.schemmel%comsys.rwth-aachen.de@localhost]

>Number:         52819
>Category:       standards
>Synopsis:       a64l should sign-extend its result when long has more than 32 bit
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    standards-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Dec 14 15:10:00 +0000 2017
>Originator:     Daniel Schemmel
>Release:        7.1
>Organization:
RWTH Aachen University
>Environment:
NetBSD   7.1 NetBSD 7.1 (GENERIC.201703111743Z) amd64
>Description:
As per POSIX, a64l is required to sign-extend its result if the type long - which the type of its result - has more than 32 bit. On amd64, the type long is a signed 64 bit integer.

Note that this relies on a slightly weird aspect of the POSIX specification: The lower 32 bit of any long are used for l64a and the behavior is unspecified if the value is negative. If long has more than 32 bit, it is never negative, as long as only the lower 32 bit are used! (Note that the glibc implementation guarantees that the behavior is never unspecified by always treating the argument as unsigned.)

This allows generating the string "zzzzz1" by giving the (positive) number 2**32-1 to l64a. This in turn ensures that the behavior of a64l is not unspecified, as the string has been generated by a call to l64a. When undoing the conversion, the lower 32 bit of the result are 0xFFFFFFFF which, when sign-extended should yield -1.

The exact wording of the POSIX 2008 standard is available at: http://pubs.opengroup.org/onlinepubs/9699919799/functions/a64l.html

This behavior was detected using Symbolic Execution techniques developed in the course of the SYMBIOSYS research project at COMSYS, RWTH Aachen University. This research is supported by the European Research Council (ERC) under the EU's Horizon 2020 Research and Innovation Programme grant agreement n. 647295 (SYMBIOSYS).
>How-To-Repeat:
Consider the following C program, which contains a test case for a64l that fails on NetBSD 7.1 when compiled for e.g. amd64:

#include <assert.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>

int main(void) {
	char const* s = l64a(0xFFFFFFFFu);
	assert(0 == strcmp(s, "zzzzz1"); // OK
	long result = a64l(s);
	long sign_extended = (long)(int32_t)result;
	assert(sign_extended == result); // FAILS
}
>Fix:
The FreeBSD implementation of a64l (https://svnweb.freebsd.org/base/head/lib/libc/stdlib/a64l.c?view=markup) is only slightly different from the NetBSD implementation (http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/stdlib/a64l.c?annotate=1.10) and does not exhibit this problem.

Alternatively, a slightly more clean approach would be to cast the result of the operation to int32_t when returning (which will explicitly show that the result is the sign-extended version of the lower 32 bit of the conversion, as required per POSIX).