[src/trunk]: src/lib/libc/gen Avoid undefined behavior in ftok(3)

[kamil <kamil%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/38a485a78d72
branches:  trunk
changeset: 991685:38a485a78d72
user:      kamil <kamil%NetBSD.org@localhost>
date:      Thu Jul 26 00:05:28 2018 +0000

description:
Avoid undefined behavior in ftok(3)

Do not change the signedness bit with a left shift operation.
Cast to unsigned integer to prevent this.

ftok.c:56:10, left shift of 123456789 by 24 places cannot be represented in type 'int'
ftok.c:56:10, left shift of 4160 by 24 places cannot be represented in type 'int'

Detected with micro-UBSan in the user mode.

diffstat:

 lib/libc/gen/ftok.c |  7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)

diffs (26 lines):

diff -r 393f44e92e35 -r 38a485a78d72 lib/libc/gen/ftok.c
--- a/lib/libc/gen/ftok.c       Wed Jul 25 23:59:08 2018 +0000
+++ b/lib/libc/gen/ftok.c       Thu Jul 26 00:05:28 2018 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: ftok.c,v 1.11 2012/03/20 16:36:05 matt Exp $   */
+/*     $NetBSD: ftok.c,v 1.12 2018/07/26 00:05:28 kamil Exp $  */
 
 /*
  * Copyright (c) 1994 SigmaSoft, Th. Lockert <tholo%sigmasoft.com@localhost>
@@ -27,7 +27,7 @@
 
 #include <sys/cdefs.h>
 #if defined(LIBC_SCCS) && !defined(lint)
-__RCSID("$NetBSD: ftok.c,v 1.11 2012/03/20 16:36:05 matt Exp $");
+__RCSID("$NetBSD: ftok.c,v 1.12 2018/07/26 00:05:28 kamil Exp $");
 #endif /* LIBC_SCCS and not lint */
 
 #include "namespace.h"
@@ -53,5 +53,6 @@
                return (key_t)-1;
 
        return (key_t)
-           (id << 24 | (st.st_dev & 0xff) << 16 | (st.st_ino & 0xffff));
+           ((unsigned int)id << 24 | (st.st_dev & 0xff) << 16 |
+            (st.st_ino & 0xffff));
 }