lib/57573: Overflow possibilities in vis(3)

[kevans%FreeBSD.org@localhost]

>Number:         57573
>Category:       lib
>Synopsis:       Overflow possibilities in vis(3)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Tue Aug 08 17:10:01 +0000 2023
>Originator:     Kyle Evans
>Release:        N/A
>Organization:
Klara, Inc.
>Environment:
FreeBSD
>Description:
We identified some overflow possibilities in vis(3) downstream in FreeBSD and wanted to alert you folks of the patch that we came up with.

Generally, there's two separate isuses:

1.) We wcrtomb() into mbst and only check that it didn't overflow *after* we already overflowed (+ same issue in the byte-by-byte fallback)

2.) The overflow check didn't account for the fact that `maxolen` includes the NUL terminator, so we could overflow when writing the NUL terminator out
>How-To-Repeat:
See tests added in referenced patch; currently we would observe that both the strnvis(3) call wouldn't fail and one byte past the end is overwritten in both of these cases, when ideally we should at the very least not write past the end of the known bounds and ultimately fail instead of simply truncating the result.
>Fix:
Patch with test included here: https://cgit.freebsd.org/src/patch/?id=2f489a509e615c46be6f7c6aa7cea161f50f18af