lib/55293: Assertion failure in libterminfo _ti_encode_buf_16

[mforney%mforney.org@localhost]

>Number:         55293
>Category:       lib
>Synopsis:       Assertion failure in libterminfo _ti_encode_buf_16
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun May 24 01:20:00 +0000 2020
>Originator:     Michael Forney
>Release:        9.9.61
>Organization:
>Environment:
NetBSD  9.99.61 NetBSD 9.99.61 (GENERIC) #0: Wed May 13 22:13:49 UTC 2020  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
After building latest libterminfo with -D_DIAGNOSTIC, an assertion failure in _ti_encode_buf_16 is triggered when running tic:
$ tic -Sx /usr/src/share/terminfo/terminfo
tic: assertion "num <= UINT16_MAX" failed: file "/usr/src/lib/libterminfo/term_private.h", line 257, function "_ti_encode_buf_16"
...

Here is the backtrace from gdb:
#0  0x0000794dd3d85998 in __diagassert13 () from /usr/lib/libc.so.12
#1  0x0000794dd48074f6 in _ti_encode_buf_16 (tbuf=0x794dd4a22e80, num=18446744073709551614) at /usr/src/lib/libterminfo/term_private.h:257
#2  0x0000794dd4808a8a in _ti_encode_buf_id_num (tbuf=0x794dd4a22e80, ind=14, num=-2, len=2) at compile.c:565
#3  0x0000794dd48092e4 in _ti_compile (cap=0x794dd47cb029 " pairs@,\n\tsetab@, setaf@, setb@, setf@, use=linux,\n", flags=21) at compile.c:
#4  0x00000001a5a024bf in process_entry (buf=0x7f7fff917660, flags=21) at tic.c:227
#5  0x00000001a5a03783 in main (argc=3, argv=0x7f7fff917728) at tic.c:673

The assertion originates in a call to _ti_encode_buf_id_num with num == CANCELLED_NUMERIC (-2). This int is converted to size_t when passed to _ti_encode_buf_32 and _ti_encode_buf_16, so results in a value of SIZE_MAX - 1, which is larger than UINT16_MAX.

I believe the intent is to store (uint16_t)-2, since that was what was done before in _ti_encode_num before it was replaced with _ti_encode_buf_id_num.
>How-To-Repeat:
Build libterminfo with -D_DIAGNOSTIC, set LIBC_DIAGASSERT=e, then run `tic -Sx /usr/src/share/terminfo/terminfo ansi`.
>Fix:
diff --git a/lib/libterminfo/compile.c b/lib/libterminfo/compile.c
index b991eb01b30f..0c89cfd12f53 100644
--- a/lib/libterminfo/compile.c
+++ b/lib/libterminfo/compile.c
@@ -560,9 +560,9 @@ _ti_encode_buf_id_num(TBUF *tbuf, int ind, int num, size_t len)
 		return 0;
 	_ti_encode_buf_16(tbuf, ind);
 	if (len == sizeof(uint32_t))
-		_ti_encode_buf_32(tbuf, num);
+		_ti_encode_buf_32(tbuf, (uint32_t)num);
 	else
-		_ti_encode_buf_16(tbuf, num);
+		_ti_encode_buf_16(tbuf, (uint16_t)num);
 	tbuf->entries++;
 	return 1;
 }