[src/netbsd-9]: src/sys Pull up following revision(s) (requested by ozaki-r i...

[martin <martin%NetBSD.org@localhost>]

details:   https://anonhg.NetBSD.org/src/rev/4709478ec357
branches:  netbsd-9
changeset: 963018:4709478ec357
user:      martin <martin%NetBSD.org@localhost>
date:      Tue Sep 24 03:10:35 2019 +0000

description:
Pull up following revision(s) (requested by ozaki-r in ticket #238):

        sys/netipsec/ipsec_output.c: revision 1.83
        sys/net/route.h: revision 1.125
        sys/netinet6/ip6_input.c: revision 1.210
        sys/netinet6/ip6_input.c: revision 1.211
        sys/net/if.c: revision 1.461
        sys/net/if_gif.h: revision 1.33
        sys/net/route.c: revision 1.220
        sys/net/route.c: revision 1.221
        sys/net/if.h: revision 1.277
        sys/netinet6/ip6_forward.c: revision 1.97
        sys/netinet/wqinput.c: revision 1.6
        sys/net/if_ipsec.h: revision 1.5
        sys/netinet6/in6_l2tp.c: revision 1.18
        sys/netinet6/in6_gif.c: revision 1.94
        sys/net/if_l2tp.h: revision 1.7
        sys/net/if_gif.c: revision 1.149
        sys/net/if_l2tp.h: revision 1.8
        sys/netinet/in_gif.c: revision 1.95
        sys/netinet/in_l2tp.c: revision 1.17
        sys/netipsec/ipsecif.c: revision 1.17
        sys/net/if_ipsec.c: revision 1.24
        sys/net/if_l2tp.c: revision 1.37
        sys/netinet/ip_input.c: revision 1.391
        sys/net/if_l2tp.c: revision 1.38
        sys/netinet/ip_input.c: revision 1.392
        sys/net/if_l2tp.c: revision 1.39

Avoid having a rtcache directly in a percpu storage

percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users.  If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.

A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.

Using rtcache, i.e., packet processing, typically involves sleepable operations
such as rwlock so we must avoid dereferencing a rtcache that is directly stored
in a percpu storage during packet processing.  Address this situation by having
just a pointer to a rtcache in a percpu storage instead.
Reviewed by knakahara@ and yamaguchi@

 -

wqinput: avoid having struct wqinput_worklist directly in a percpu storage

percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users.  If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.

A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.

Input handlers of wqinput normally involves sleepable operations so we must
avoid dereferencing a percpu data (struct wqinput_worklist) after executing
an input handler.  Address this situation by having just a pointer to the data
in a percpu storage instead.
Reviewed by knakahara@ and yamaguchi@

 -

Add missing #include <sys/kmem.h>

 -

Divide Tx context of l2tp(4) to improve performance.

It seems l2tp(4) call path is too long for instruction cache. So, dividing
l2tp(4) Tx context improves CPU use efficiency.

After this commit, l2tp(4) throughput gains 10% on my machine(Atom C3000).

 -

Apply some missing changes lost on the previous commit

 -

Avoid having a rtcache directly in a percpu storage for tunnel protocols.
percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users.  If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.

A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.

Using rtcache, i.e., packet processing, typically involves sleepable operations
such as rwlock so we must avoid dereferencing a rtcache that is directly stored
in a percpu storage during packet processing.  Address this situation by having
just a pointer to a rtcache in a percpu storage instead.

Reviewed by ozaki-r@ and yamaguchi@

 -

l2tp(4): avoid having struct ifqueue directly in a percpu storage.
percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users.  If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.

A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.

Tx processing of l2tp(4) uses normally involves sleepable operations so we
must avoid dereferencing a percpu data (struct ifqueue) after executing Tx
processing.  Address this situation by having just a pointer to the data in
a percpu storage instead.

Reviewed by ozaki-r@ and yamaguchi@

diffstat:

 sys/net/if.c                |   61 ++++++++++-
 sys/net/if.h                |   29 +++++-
 sys/net/if_gif.c            |   39 +------
 sys/net/if_gif.h            |   15 +--
 sys/net/if_ipsec.c          |   31 +----
 sys/net/if_ipsec.h          |   11 +-
 sys/net/if_l2tp.c           |  242 ++++++++++++++++++++++++++++---------------
 sys/net/if_l2tp.h           |   14 +-
 sys/net/route.c             |   28 ++++-
 sys/net/route.h             |   21 +++-
 sys/netinet/in_gif.c        |   33 ++---
 sys/netinet/in_l2tp.c       |   30 ++--
 sys/netinet/ip_input.c      |   24 ++--
 sys/netinet/wqinput.c       |   38 +++++-
 sys/netinet6/in6_gif.c      |   46 +++----
 sys/netinet6/in6_l2tp.c     |   29 ++--
 sys/netinet6/ip6_forward.c  |    8 +-
 sys/netinet6/ip6_input.c    |   22 ++--
 sys/netipsec/ipsec_output.c |   10 +-
 sys/netipsec/ipsecif.c      |   56 +++------
 20 files changed, 456 insertions(+), 331 deletions(-)

diffs (truncated from 1751 to 300 lines):

diff -r ef58d64d1f34 -r 4709478ec357 sys/net/if.c
--- a/sys/net/if.c      Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if.c      Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: if.c,v 1.457.2.1 2019/08/19 16:10:35 martin Exp $      */
+/*     $NetBSD: if.c,v 1.457.2.2 2019/09/24 03:10:35 martin Exp $      */
 
 /*-
  * Copyright (c) 1999, 2000, 2001, 2008 The NetBSD Foundation, Inc.
@@ -90,7 +90,7 @@
  */
 
 #include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if.c,v 1.457.2.1 2019/08/19 16:10:35 martin Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if.c,v 1.457.2.2 2019/09/24 03:10:35 martin Exp $");
 
 #if defined(_KERNEL_OPT)
 #include "opt_inet.h"
@@ -2906,6 +2906,63 @@
        return 0;
 }
 
+static void
+if_tunnel_ro_init_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
+{
+       struct tunnel_ro *tro = p;
+
+       tro->tr_ro = kmem_zalloc(sizeof(*tro->tr_ro), KM_SLEEP);
+       tro->tr_lock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NONE);
+}
+
+percpu_t *
+if_tunnel_alloc_ro_percpu(void)
+{
+       percpu_t *ro_percpu;
+
+       ro_percpu = percpu_alloc(sizeof(struct tunnel_ro));
+       percpu_foreach(ro_percpu, if_tunnel_ro_init_pc, NULL);
+
+       return ro_percpu;
+}
+
+static void
+if_tunnel_ro_fini_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
+{
+       struct tunnel_ro *tro = p;
+
+       rtcache_free(tro->tr_ro);
+       kmem_free(tro->tr_ro, sizeof(*tro->tr_ro));
+
+       mutex_obj_free(tro->tr_lock);
+}
+
+void
+if_tunnel_free_ro_percpu(percpu_t *ro_percpu)
+{
+
+       percpu_foreach(ro_percpu, if_tunnel_ro_fini_pc, NULL);
+       percpu_free(ro_percpu, sizeof(struct tunnel_ro));
+}
+
+
+static void
+if_tunnel_rtcache_free_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
+{
+       struct tunnel_ro *tro = p;
+
+       mutex_enter(tro->tr_lock);
+       rtcache_free(tro->tr_ro);
+       mutex_exit(tro->tr_lock);
+}
+
+void if_tunnel_ro_percpu_rtcache_free(percpu_t *ro_percpu)
+{
+
+       percpu_foreach(ro_percpu, if_tunnel_rtcache_free_pc, NULL);
+}
+
+
 /* common */
 int
 ifioctl_common(struct ifnet *ifp, u_long cmd, void *data)
diff -r ef58d64d1f34 -r 4709478ec357 sys/net/if.h
--- a/sys/net/if.h      Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if.h      Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: if.h,v 1.274 2019/07/04 02:44:25 ozaki-r Exp $ */
+/*     $NetBSD: if.h,v 1.274.2.1 2019/09/24 03:10:35 martin Exp $      */
 
 /*-
  * Copyright (c) 1999, 2000, 2001 The NetBSD Foundation, Inc.
@@ -1121,6 +1121,33 @@
 #define        if_release      if_put
 
 int if_tunnel_check_nesting(struct ifnet *, struct mbuf *, int);
+percpu_t *if_tunnel_alloc_ro_percpu(void);
+void if_tunnel_free_ro_percpu(percpu_t *);
+void if_tunnel_ro_percpu_rtcache_free(percpu_t *);
+
+struct tunnel_ro {
+       struct route *tr_ro;
+       kmutex_t *tr_lock;
+};
+
+static inline void
+if_tunnel_get_ro(percpu_t *ro_percpu, struct route **ro, kmutex_t **lock)
+{
+       struct tunnel_ro *tro;
+
+       tro = percpu_getref(ro_percpu);
+       *ro = tro->tr_ro;
+       *lock = tro->tr_lock;
+       mutex_enter(*lock);
+}
+
+static inline void
+if_tunnel_put_ro(percpu_t *ro_percpu, kmutex_t *lock)
+{
+
+       mutex_exit(lock);
+       percpu_putref(ro_percpu);
+}
 
 static __inline if_index_t
 if_get_index(const struct ifnet *ifp)
diff -r ef58d64d1f34 -r 4709478ec357 sys/net/if_gif.c
--- a/sys/net/if_gif.c  Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if_gif.c  Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: if_gif.c,v 1.148 2019/06/25 12:30:50 msaitoh Exp $     */
+/*     $NetBSD: if_gif.c,v 1.148.2.1 2019/09/24 03:10:35 martin Exp $  */
 /*     $KAME: if_gif.c,v 1.76 2001/08/20 02:01:02 kjc Exp $    */
 
 /*
@@ -31,7 +31,7 @@
  */
 
 #include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if_gif.c,v 1.148 2019/06/25 12:30:50 msaitoh Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if_gif.c,v 1.148.2.1 2019/09/24 03:10:35 martin Exp $");
 
 #ifdef _KERNEL_OPT
 #include "opt_inet.h"
@@ -104,9 +104,6 @@
 
 struct psref_class *gv_psref_class __read_mostly;
 
-static void    gif_ro_init_pc(void *, void *, struct cpu_info *);
-static void    gif_ro_fini_pc(void *, void *, struct cpu_info *);
-
 static int     gifattach0(struct gif_softc *);
 static int     gif_output(struct ifnet *, struct mbuf *,
                           const struct sockaddr *, const struct rtentry *);
@@ -271,8 +268,7 @@
        mutex_init(&sc->gif_lock, MUTEX_DEFAULT, IPL_NONE);
        sc->gif_psz = pserialize_create();
 
-       sc->gif_ro_percpu = percpu_alloc(sizeof(struct gif_ro));
-       percpu_foreach(sc->gif_ro_percpu, gif_ro_init_pc, NULL);
+       sc->gif_ro_percpu = if_tunnel_alloc_ro_percpu();
        mutex_enter(&gif_softcs.lock);
        LIST_INSERT_HEAD(&gif_softcs.list, sc, gif_list);
        mutex_exit(&gif_softcs.lock);
@@ -309,32 +305,6 @@
        return 0;
 }
 
-static void
-gif_ro_init_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
-{
-       struct gif_ro *gro = p;
-
-       gro->gr_lock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NONE);
-}
-
-static void
-gif_ro_fini_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
-{
-       struct gif_ro *gro = p;
-
-       rtcache_free(&gro->gr_ro);
-
-       mutex_obj_free(gro->gr_lock);
-}
-
-void
-gif_rtcache_free_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
-{
-       struct gif_ro *gro = p;
-
-       rtcache_free(&gro->gr_ro);
-}
-
 static int
 gif_clone_destroy(struct ifnet *ifp)
 {
@@ -347,8 +317,7 @@
        bpf_detach(ifp);
        if_detach(ifp);
 
-       percpu_foreach(sc->gif_ro_percpu, gif_ro_fini_pc, NULL);
-       percpu_free(sc->gif_ro_percpu, sizeof(struct gif_ro));
+       if_tunnel_free_ro_percpu(sc->gif_ro_percpu);
 
        pserialize_destroy(sc->gif_psz);
        mutex_destroy(&sc->gif_lock);
diff -r ef58d64d1f34 -r 4709478ec357 sys/net/if_gif.h
--- a/sys/net/if_gif.h  Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if_gif.h  Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: if_gif.h,v 1.32 2018/10/19 00:12:56 knakahara Exp $    */
+/*     $NetBSD: if_gif.h,v 1.32.4.1 2019/09/24 03:10:35 martin Exp $   */
 /*     $KAME: if_gif.h,v 1.23 2001/07/27 09:21:42 itojun Exp $ */
 
 /*
@@ -55,11 +55,6 @@
 
 struct encaptab;
 
-struct gif_ro {
-       struct route gr_ro;
-       kmutex_t *gr_lock;
-};
-
 struct gif_variant {
        struct gif_softc *gv_softc;
        struct sockaddr *gv_psrc; /* Physical src addr */
@@ -73,7 +68,7 @@
 
 struct gif_softc {
        struct ifnet    gif_if;         /* common area - must be at the top */
-       percpu_t *gif_ro_percpu;        /* struct gif_ro */
+       percpu_t *gif_ro_percpu;        /* struct tunnel_ro */
        struct gif_variant *gif_var;    /*
                                         * reader must use gif_getref_variant()
                                         * instead of direct dereference.
@@ -131,8 +126,6 @@
 /* Prototypes */
 void   gif_input(struct mbuf *, int, struct ifnet *);
 
-void   gif_rtcache_free_pc(void *, void *, struct cpu_info *);
-
 #ifdef GIF_ENCAPCHECK
 int    gif_encapcheck(struct mbuf *, int, int, void *);
 #endif
@@ -147,8 +140,8 @@
  *   - gif_var->gv_psref for reader
  *       gif_softc->gif_var is used for variant values while the gif tunnel
  *       exists.
- * + Each CPU's gif_ro.gr_ro of gif_ro_percpu are protected by
- *   percpu'ed gif_ro.gr_lock.
+ * + Each CPU's tunnel_ro.tr_ro of gif_ro_percpu are protected by
+ *   percpu'ed tunnel_ro.tr_lock.
  *
  * Locking order:
  *     - encap_lock => gif_softc->gif_lock => gif_softcs.lock
diff -r ef58d64d1f34 -r 4709478ec357 sys/net/if_ipsec.c
--- a/sys/net/if_ipsec.c        Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if_ipsec.c        Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/*     $NetBSD: if_ipsec.c,v 1.22 2019/06/25 12:30:50 msaitoh Exp $  */
+/*     $NetBSD: if_ipsec.c,v 1.22.2.1 2019/09/24 03:10:35 martin Exp $  */
 
 /*
  * Copyright (c) 2017 Internet Initiative Japan Inc.
@@ -27,7 +27,7 @@
  */
 
 #include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if_ipsec.c,v 1.22 2019/06/25 12:30:50 msaitoh Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if_ipsec.c,v 1.22.2.1 2019/09/24 03:10:35 martin Exp $");
 
 #ifdef _KERNEL_OPT
 #include "opt_inet.h"
@@ -80,9 +80,6 @@
 #include <netipsec/ipsec.h>
 #include <netipsec/ipsecif.h>
 
-static void if_ipsec_ro_init_pc(void *, void *, struct cpu_info *);
-static void if_ipsec_ro_fini_pc(void *, void *, struct cpu_info *);
-
 static int if_ipsec_clone_create(struct if_clone *, int);
 static int if_ipsec_clone_destroy(struct ifnet *);
 
@@ -182,8 +179,7 @@
        sc->ipsec_var = var;
        mutex_init(&sc->ipsec_lock, MUTEX_DEFAULT, IPL_NONE);
        sc->ipsec_psz = pserialize_create();
-       sc->ipsec_ro_percpu = percpu_alloc(sizeof(struct ipsec_ro));
-       percpu_foreach(sc->ipsec_ro_percpu, if_ipsec_ro_init_pc, NULL);
+       sc->ipsec_ro_percpu = if_tunnel_alloc_ro_percpu();
 
        mutex_enter(&ipsec_softcs.lock);
        LIST_INSERT_HEAD(&ipsec_softcs.list, sc, ipsec_list);
@@ -213,24 +209,6 @@
        if_register(&sc->ipsec_if);
 }
 
-static void
-if_ipsec_ro_init_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)