Source-Changes-HG archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
[src/netbsd-9]: src/sys Pull up following revision(s) (requested by ozaki-r i...
details: https://anonhg.NetBSD.org/src/rev/027f1e12f822
branches: netbsd-9
changeset: 1000889:027f1e12f822
user: martin <martin%NetBSD.org@localhost>
date: Tue Sep 24 03:10:35 2019 +0000
description:
Pull up following revision(s) (requested by ozaki-r in ticket #238):
sys/netipsec/ipsec_output.c: revision 1.83
sys/net/route.h: revision 1.125
sys/netinet6/ip6_input.c: revision 1.210
sys/netinet6/ip6_input.c: revision 1.211
sys/net/if.c: revision 1.461
sys/net/if_gif.h: revision 1.33
sys/net/route.c: revision 1.220
sys/net/route.c: revision 1.221
sys/net/if.h: revision 1.277
sys/netinet6/ip6_forward.c: revision 1.97
sys/netinet/wqinput.c: revision 1.6
sys/net/if_ipsec.h: revision 1.5
sys/netinet6/in6_l2tp.c: revision 1.18
sys/netinet6/in6_gif.c: revision 1.94
sys/net/if_l2tp.h: revision 1.7
sys/net/if_gif.c: revision 1.149
sys/net/if_l2tp.h: revision 1.8
sys/netinet/in_gif.c: revision 1.95
sys/netinet/in_l2tp.c: revision 1.17
sys/netipsec/ipsecif.c: revision 1.17
sys/net/if_ipsec.c: revision 1.24
sys/net/if_l2tp.c: revision 1.37
sys/netinet/ip_input.c: revision 1.391
sys/net/if_l2tp.c: revision 1.38
sys/netinet/ip_input.c: revision 1.392
sys/net/if_l2tp.c: revision 1.39
Avoid having a rtcache directly in a percpu storage
percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users. If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.
A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.
Using rtcache, i.e., packet processing, typically involves sleepable operations
such as rwlock so we must avoid dereferencing a rtcache that is directly stored
in a percpu storage during packet processing. Address this situation by having
just a pointer to a rtcache in a percpu storage instead.
Reviewed by knakahara@ and yamaguchi@
-
wqinput: avoid having struct wqinput_worklist directly in a percpu storage
percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users. If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.
A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.
Input handlers of wqinput normally involves sleepable operations so we must
avoid dereferencing a percpu data (struct wqinput_worklist) after executing
an input handler. Address this situation by having just a pointer to the data
in a percpu storage instead.
Reviewed by knakahara@ and yamaguchi@
-
Add missing #include <sys/kmem.h>
-
Divide Tx context of l2tp(4) to improve performance.
It seems l2tp(4) call path is too long for instruction cache. So, dividing
l2tp(4) Tx context improves CPU use efficiency.
After this commit, l2tp(4) throughput gains 10% on my machine(Atom C3000).
-
Apply some missing changes lost on the previous commit
-
Avoid having a rtcache directly in a percpu storage for tunnel protocols.
percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users. If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.
A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.
Using rtcache, i.e., packet processing, typically involves sleepable operations
such as rwlock so we must avoid dereferencing a rtcache that is directly stored
in a percpu storage during packet processing. Address this situation by having
just a pointer to a rtcache in a percpu storage instead.
Reviewed by ozaki-r@ and yamaguchi@
-
l2tp(4): avoid having struct ifqueue directly in a percpu storage.
percpu(9) has a certain memory storage for each CPU and provides it by the piece
to users. If the storages went short, percpu(9) enlarges them by allocating new
larger memory areas, replacing old ones with them and destroying the old ones.
A percpu storage referenced by a pointer gotten via percpu_getref can be
destroyed by the mechanism after a running thread sleeps even if percpu_putref
has not been called.
Tx processing of l2tp(4) uses normally involves sleepable operations so we
must avoid dereferencing a percpu data (struct ifqueue) after executing Tx
processing. Address this situation by having just a pointer to the data in
a percpu storage instead.
Reviewed by ozaki-r@ and yamaguchi@
diffstat:
sys/net/if.c | 61 ++++++++++-
sys/net/if.h | 29 +++++-
sys/net/if_gif.c | 39 +------
sys/net/if_gif.h | 15 +--
sys/net/if_ipsec.c | 31 +----
sys/net/if_ipsec.h | 11 +-
sys/net/if_l2tp.c | 242 ++++++++++++++++++++++++++++---------------
sys/net/if_l2tp.h | 14 +-
sys/net/route.c | 28 ++++-
sys/net/route.h | 21 +++-
sys/netinet/in_gif.c | 33 ++---
sys/netinet/in_l2tp.c | 30 ++--
sys/netinet/ip_input.c | 24 ++--
sys/netinet/wqinput.c | 38 +++++-
sys/netinet6/in6_gif.c | 46 +++----
sys/netinet6/in6_l2tp.c | 29 ++--
sys/netinet6/ip6_forward.c | 8 +-
sys/netinet6/ip6_input.c | 22 ++--
sys/netipsec/ipsec_output.c | 10 +-
sys/netipsec/ipsecif.c | 56 +++------
20 files changed, 456 insertions(+), 331 deletions(-)
diffs (truncated from 1751 to 300 lines):
diff -r 2f3aa3b37dbb -r 027f1e12f822 sys/net/if.c
--- a/sys/net/if.c Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if.c Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if.c,v 1.457.2.1 2019/08/19 16:10:35 martin Exp $ */
+/* $NetBSD: if.c,v 1.457.2.2 2019/09/24 03:10:35 martin Exp $ */
/*-
* Copyright (c) 1999, 2000, 2001, 2008 The NetBSD Foundation, Inc.
@@ -90,7 +90,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if.c,v 1.457.2.1 2019/08/19 16:10:35 martin Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if.c,v 1.457.2.2 2019/09/24 03:10:35 martin Exp $");
#if defined(_KERNEL_OPT)
#include "opt_inet.h"
@@ -2906,6 +2906,63 @@
return 0;
}
+static void
+if_tunnel_ro_init_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
+{
+ struct tunnel_ro *tro = p;
+
+ tro->tr_ro = kmem_zalloc(sizeof(*tro->tr_ro), KM_SLEEP);
+ tro->tr_lock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NONE);
+}
+
+percpu_t *
+if_tunnel_alloc_ro_percpu(void)
+{
+ percpu_t *ro_percpu;
+
+ ro_percpu = percpu_alloc(sizeof(struct tunnel_ro));
+ percpu_foreach(ro_percpu, if_tunnel_ro_init_pc, NULL);
+
+ return ro_percpu;
+}
+
+static void
+if_tunnel_ro_fini_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
+{
+ struct tunnel_ro *tro = p;
+
+ rtcache_free(tro->tr_ro);
+ kmem_free(tro->tr_ro, sizeof(*tro->tr_ro));
+
+ mutex_obj_free(tro->tr_lock);
+}
+
+void
+if_tunnel_free_ro_percpu(percpu_t *ro_percpu)
+{
+
+ percpu_foreach(ro_percpu, if_tunnel_ro_fini_pc, NULL);
+ percpu_free(ro_percpu, sizeof(struct tunnel_ro));
+}
+
+
+static void
+if_tunnel_rtcache_free_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
+{
+ struct tunnel_ro *tro = p;
+
+ mutex_enter(tro->tr_lock);
+ rtcache_free(tro->tr_ro);
+ mutex_exit(tro->tr_lock);
+}
+
+void if_tunnel_ro_percpu_rtcache_free(percpu_t *ro_percpu)
+{
+
+ percpu_foreach(ro_percpu, if_tunnel_rtcache_free_pc, NULL);
+}
+
+
/* common */
int
ifioctl_common(struct ifnet *ifp, u_long cmd, void *data)
diff -r 2f3aa3b37dbb -r 027f1e12f822 sys/net/if.h
--- a/sys/net/if.h Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if.h Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if.h,v 1.274 2019/07/04 02:44:25 ozaki-r Exp $ */
+/* $NetBSD: if.h,v 1.274.2.1 2019/09/24 03:10:35 martin Exp $ */
/*-
* Copyright (c) 1999, 2000, 2001 The NetBSD Foundation, Inc.
@@ -1121,6 +1121,33 @@
#define if_release if_put
int if_tunnel_check_nesting(struct ifnet *, struct mbuf *, int);
+percpu_t *if_tunnel_alloc_ro_percpu(void);
+void if_tunnel_free_ro_percpu(percpu_t *);
+void if_tunnel_ro_percpu_rtcache_free(percpu_t *);
+
+struct tunnel_ro {
+ struct route *tr_ro;
+ kmutex_t *tr_lock;
+};
+
+static inline void
+if_tunnel_get_ro(percpu_t *ro_percpu, struct route **ro, kmutex_t **lock)
+{
+ struct tunnel_ro *tro;
+
+ tro = percpu_getref(ro_percpu);
+ *ro = tro->tr_ro;
+ *lock = tro->tr_lock;
+ mutex_enter(*lock);
+}
+
+static inline void
+if_tunnel_put_ro(percpu_t *ro_percpu, kmutex_t *lock)
+{
+
+ mutex_exit(lock);
+ percpu_putref(ro_percpu);
+}
static __inline if_index_t
if_get_index(const struct ifnet *ifp)
diff -r 2f3aa3b37dbb -r 027f1e12f822 sys/net/if_gif.c
--- a/sys/net/if_gif.c Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if_gif.c Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if_gif.c,v 1.148 2019/06/25 12:30:50 msaitoh Exp $ */
+/* $NetBSD: if_gif.c,v 1.148.2.1 2019/09/24 03:10:35 martin Exp $ */
/* $KAME: if_gif.c,v 1.76 2001/08/20 02:01:02 kjc Exp $ */
/*
@@ -31,7 +31,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if_gif.c,v 1.148 2019/06/25 12:30:50 msaitoh Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if_gif.c,v 1.148.2.1 2019/09/24 03:10:35 martin Exp $");
#ifdef _KERNEL_OPT
#include "opt_inet.h"
@@ -104,9 +104,6 @@
struct psref_class *gv_psref_class __read_mostly;
-static void gif_ro_init_pc(void *, void *, struct cpu_info *);
-static void gif_ro_fini_pc(void *, void *, struct cpu_info *);
-
static int gifattach0(struct gif_softc *);
static int gif_output(struct ifnet *, struct mbuf *,
const struct sockaddr *, const struct rtentry *);
@@ -271,8 +268,7 @@
mutex_init(&sc->gif_lock, MUTEX_DEFAULT, IPL_NONE);
sc->gif_psz = pserialize_create();
- sc->gif_ro_percpu = percpu_alloc(sizeof(struct gif_ro));
- percpu_foreach(sc->gif_ro_percpu, gif_ro_init_pc, NULL);
+ sc->gif_ro_percpu = if_tunnel_alloc_ro_percpu();
mutex_enter(&gif_softcs.lock);
LIST_INSERT_HEAD(&gif_softcs.list, sc, gif_list);
mutex_exit(&gif_softcs.lock);
@@ -309,32 +305,6 @@
return 0;
}
-static void
-gif_ro_init_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
-{
- struct gif_ro *gro = p;
-
- gro->gr_lock = mutex_obj_alloc(MUTEX_DEFAULT, IPL_NONE);
-}
-
-static void
-gif_ro_fini_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
-{
- struct gif_ro *gro = p;
-
- rtcache_free(&gro->gr_ro);
-
- mutex_obj_free(gro->gr_lock);
-}
-
-void
-gif_rtcache_free_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
-{
- struct gif_ro *gro = p;
-
- rtcache_free(&gro->gr_ro);
-}
-
static int
gif_clone_destroy(struct ifnet *ifp)
{
@@ -347,8 +317,7 @@
bpf_detach(ifp);
if_detach(ifp);
- percpu_foreach(sc->gif_ro_percpu, gif_ro_fini_pc, NULL);
- percpu_free(sc->gif_ro_percpu, sizeof(struct gif_ro));
+ if_tunnel_free_ro_percpu(sc->gif_ro_percpu);
pserialize_destroy(sc->gif_psz);
mutex_destroy(&sc->gif_lock);
diff -r 2f3aa3b37dbb -r 027f1e12f822 sys/net/if_gif.h
--- a/sys/net/if_gif.h Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if_gif.h Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if_gif.h,v 1.32 2018/10/19 00:12:56 knakahara Exp $ */
+/* $NetBSD: if_gif.h,v 1.32.4.1 2019/09/24 03:10:35 martin Exp $ */
/* $KAME: if_gif.h,v 1.23 2001/07/27 09:21:42 itojun Exp $ */
/*
@@ -55,11 +55,6 @@
struct encaptab;
-struct gif_ro {
- struct route gr_ro;
- kmutex_t *gr_lock;
-};
-
struct gif_variant {
struct gif_softc *gv_softc;
struct sockaddr *gv_psrc; /* Physical src addr */
@@ -73,7 +68,7 @@
struct gif_softc {
struct ifnet gif_if; /* common area - must be at the top */
- percpu_t *gif_ro_percpu; /* struct gif_ro */
+ percpu_t *gif_ro_percpu; /* struct tunnel_ro */
struct gif_variant *gif_var; /*
* reader must use gif_getref_variant()
* instead of direct dereference.
@@ -131,8 +126,6 @@
/* Prototypes */
void gif_input(struct mbuf *, int, struct ifnet *);
-void gif_rtcache_free_pc(void *, void *, struct cpu_info *);
-
#ifdef GIF_ENCAPCHECK
int gif_encapcheck(struct mbuf *, int, int, void *);
#endif
@@ -147,8 +140,8 @@
* - gif_var->gv_psref for reader
* gif_softc->gif_var is used for variant values while the gif tunnel
* exists.
- * + Each CPU's gif_ro.gr_ro of gif_ro_percpu are protected by
- * percpu'ed gif_ro.gr_lock.
+ * + Each CPU's tunnel_ro.tr_ro of gif_ro_percpu are protected by
+ * percpu'ed tunnel_ro.tr_lock.
*
* Locking order:
* - encap_lock => gif_softc->gif_lock => gif_softcs.lock
diff -r 2f3aa3b37dbb -r 027f1e12f822 sys/net/if_ipsec.c
--- a/sys/net/if_ipsec.c Tue Sep 24 03:02:01 2019 +0000
+++ b/sys/net/if_ipsec.c Tue Sep 24 03:10:35 2019 +0000
@@ -1,4 +1,4 @@
-/* $NetBSD: if_ipsec.c,v 1.22 2019/06/25 12:30:50 msaitoh Exp $ */
+/* $NetBSD: if_ipsec.c,v 1.22.2.1 2019/09/24 03:10:35 martin Exp $ */
/*
* Copyright (c) 2017 Internet Initiative Japan Inc.
@@ -27,7 +27,7 @@
*/
#include <sys/cdefs.h>
-__KERNEL_RCSID(0, "$NetBSD: if_ipsec.c,v 1.22 2019/06/25 12:30:50 msaitoh Exp $");
+__KERNEL_RCSID(0, "$NetBSD: if_ipsec.c,v 1.22.2.1 2019/09/24 03:10:35 martin Exp $");
#ifdef _KERNEL_OPT
#include "opt_inet.h"
@@ -80,9 +80,6 @@
#include <netipsec/ipsec.h>
#include <netipsec/ipsecif.h>
-static void if_ipsec_ro_init_pc(void *, void *, struct cpu_info *);
-static void if_ipsec_ro_fini_pc(void *, void *, struct cpu_info *);
-
static int if_ipsec_clone_create(struct if_clone *, int);
static int if_ipsec_clone_destroy(struct ifnet *);
@@ -182,8 +179,7 @@
sc->ipsec_var = var;
mutex_init(&sc->ipsec_lock, MUTEX_DEFAULT, IPL_NONE);
sc->ipsec_psz = pserialize_create();
- sc->ipsec_ro_percpu = percpu_alloc(sizeof(struct ipsec_ro));
- percpu_foreach(sc->ipsec_ro_percpu, if_ipsec_ro_init_pc, NULL);
+ sc->ipsec_ro_percpu = if_tunnel_alloc_ro_percpu();
mutex_enter(&ipsec_softcs.lock);
LIST_INSERT_HEAD(&ipsec_softcs.list, sc, ipsec_list);
@@ -213,24 +209,6 @@
if_register(&sc->ipsec_if);
}
-static void
-if_ipsec_ro_init_pc(void *p, void *arg __unused, struct cpu_info *ci __unused)
Home |
Main Index |
Thread Index |
Old Index