misc/59198: Reject negative input in strtou(3), reporting ERANGE

[alx%kernel.org@localhost]

>Number:         59198
>Category:       misc
>Synopsis:       Reject negative input in strtou(3), reporting ERANGE
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    misc-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Mar 19 23:25:00 +0000 2025
>Originator:     Alejandro Colomar
>Release:        10.1
>Organization:
Linux man-pages project, C Committee (WG14), shadow-utils
>Environment:
>Description:
strtou(3), just like the standard strtoumax(3) family of functions, first converts negative values into huge positive values, and then performs the range checks.

This is behavior that most programmers don't even know, less expect.

In every use case of this API, I've always wanted to reject negative input.  In the standard strtoumax(3), and also in the strtou(3) API.  However, I think it is more relevant in strtou(3), because we have explicit range checks.  If I say the minimum acceptable value is 0, then -1 is certainly not an acceptable value, not even converted via modulo arithmetics into UINTMAX_MAX.  It should result in ERANGE.

I have developed my own wrapper, strtou_noneg():

uintmax_t
strtou_noneg(const char *s, char **restrict endp, int base,
    uintmax_t min, uintmax_t max, int *restrict status)
{
	int  st;

	if (status == NULL)
		status = &st;
	if (strtoi(s, endp, base, 0, 1, status) == 0 && *status == ERANGE)
		return min;

	return strtou(s, endp, base, min, max, status);
}


and I always call that, because the behavior of strtou(3) is just evil.  Would you mind reforming strtou(3) to behave like that, rejecting negative input?  I suspect there are no real users of that misfeature, and probably many callers will silently have their bugs fixed.

While discussing standardization of this API, we found some bugs in GNU gettext where the author was unaware that strtoul(3) had this evil behavior.  There are many more programmers that are unaware of this.
>How-To-Repeat:
strtou("-1", NULL, 0, 0, UINTMAX_MAX, &status);

// status will be 0, but should be ERANGE
>Fix:
I have developed my own wrapper, strtou_noneg():

uintmax_t
strtou_noneg(const char *s, char **restrict endp, int base,
    uintmax_t min, uintmax_t max, int *restrict status)
{
	int  st;

	if (status == NULL)
		status = &st;
	if (strtoi(s, endp, base, 0, 1, status) == 0 && *status == ERANGE)
		return min;

	return strtou(s, endp, base, min, max, status);
}

This should be the actual implementation of strtou(3).