[src/trunk]: src/sys/kern Handle p->p_mqueue_cnt in a symmetric manner.

[jakllsch <jakllsch%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/46a2fae111d5
branches:  trunk
changeset: 363990:46a2fae111d5
user:      jakllsch <jakllsch%NetBSD.org@localhost>
date:      Sun Aug 19 15:10:23 2018 +0000

description:
Handle p->p_mqueue_cnt in a symmetric manner.

While this change does mean that each open mqd_t counts against the
limit, it also means that p_mqueue_cnt will never wrap below 0.

Discovered by the Rust nix crate test suite.  It opens the same mq twice
(once for writes, once for reads), and then proceeds to close each of
them, which resulted in p_mqueue_cnt wrapping to (u_int)-1, preventing
the process from opening any more mqueues, and all subsequent
mq(3)-related tests to fail.

diffstat:

 sys/kern/sys_mqueue.c |  25 ++++++++++---------------
 1 files changed, 10 insertions(+), 15 deletions(-)

diffs (60 lines):

diff -r ba7b18ddec8a -r 46a2fae111d5 sys/kern/sys_mqueue.c
--- a/sys/kern/sys_mqueue.c     Sun Aug 19 14:50:24 2018 +0000
+++ b/sys/kern/sys_mqueue.c     Sun Aug 19 15:10:23 2018 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: sys_mqueue.c,v 1.42 2018/07/04 17:50:18 kamil Exp $    */
+/*     $NetBSD: sys_mqueue.c,v 1.43 2018/08/19 15:10:23 jakllsch Exp $ */
 
 /*
  * Copyright (c) 2007-2011 Mindaugas Rasiukevicius <rmind at NetBSD org>
@@ -43,7 +43,7 @@
  */
 
 #include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: sys_mqueue.c,v 1.42 2018/07/04 17:50:18 kamil Exp $");
+__KERNEL_RCSID(0, "$NetBSD: sys_mqueue.c,v 1.43 2018/08/19 15:10:23 jakllsch Exp $");
 
 #include <sys/param.h>
 #include <sys/types.h>
@@ -431,11 +431,6 @@
        mqueue_t *mq;
        u_int i;
 
-       /* Pre-check the limit. */
-       if (p->p_mqueue_cnt >= mq_open_max) {
-               return EMFILE;
-       }
-
        /* Empty name is invalid. */
        if (name[0] == '\0') {
                return EINVAL;
@@ -516,6 +511,14 @@
                kmem_free(name, MQ_NAMELEN);
                return error;
        }
+
+       /* Account and check for the limit. */
+       if (atomic_inc_uint_nv(&p->p_mqueue_cnt) > mq_open_max) {
+               atomic_dec_uint(&p->p_mqueue_cnt);
+               error = EMFILE;
+               goto err;
+       }
+
        fp->f_type = DTYPE_MQUEUE;
        fp->f_flag = FFLAGS(oflag) & (FREAD | FWRITE);
        fp->f_ops = &mqops;
@@ -562,14 +565,6 @@
                        goto err;
                }
 
-               /* Account and check for the limit. */
-               if (atomic_inc_uint_nv(&p->p_mqueue_cnt) > mq_open_max) {
-                       mutex_exit(&mqlist_lock);
-                       atomic_dec_uint(&p->p_mqueue_cnt);
-                       error = EMFILE;
-                       goto err;
-               }
-
                /* Initial timestamps. */
                mq = mq_new;
                getnanotime(&mq->mq_btime);