Re: NetBSD 5.1 TCP performance issue (lots of ACK)

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Thu, Oct 27, 2011 at 12:00:33PM -0400, Thor Lancelot Simon wrote:
> It's possible this has to do with the interrupt moderation tuning.  I
> believe we've been pending the checkin of better values than the ones
> I worked out from the documentation for quite some time -- there were
> highly unobvious performance effects with small buffers.  Simon did
> a bunch of testing and concluded, as I recall, that the values used
> by Intel in the Linux driver were "magic" and that we should use
> those, not mine.
> 
> If this hasn't been adjusted to match the Linux driver, you might
> want to take a quick look at the values it uses and see whether
> they yield better small-buffer performance in your case.

I looked quickly at this and came up with the attached patch.

With this (installed on both NetBSD hosts) I get mittiged results:
- the NetBSD client against the linux server gets degranded and unstable
  performances several runs gives large variations in speed
- the NetBSD client against the NetBSD server gets better performances
  in average (but still not in the 90MB range) and also with large
  variations between runs
- the linux client against the NetBSD server gets a little boost and
  the speed is stll stable between runs
- ttcp performances between NetBSD hosts gets a little boost too,
  and the speed is stll stable between runs

But I do get Ierrs on both NetBSD hosts now, with the ttcp or glusterfs
test. I don't know where these errors comes from. Linux has no errors.
I don't think it's wm_add_rxbuf(), netstat -m and vmstat -m shows
no issues with mbuf allocations.
So I guess these are errors at the adapter level, we may need to change
more things to match these values.
Also, linux seems to be using more advanced features for these adapters,
this is something we may have to look at too.

> 
> Also it may be worth checking we are not doing TSO twice, once in
> hardware and once in software.  There was a bug a long time ago
> that could cause that but last time I looked I determined by setting
> breakpoints in the relevant segmentation functions that it was,
> as far as I could tell, fixed.

This would affect the linux client/server against the NetBSD server/client
as well, isn't it ?

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Index: if_wm.c
===================================================================
RCS file: /cvsroot/src/sys/dev/pci/if_wm.c,v
retrieving revision 1.162.4.15
diff -u -p -u -r1.162.4.15 if_wm.c
--- if_wm.c     7 Mar 2011 04:14:19 -0000       1.162.4.15
+++ if_wm.c     27 Oct 2011 16:56:20 -0000
@@ -181,7 +181,7 @@ int wm_debug = WM_DEBUG_TX|WM_DEBUG_RX|W
  * packet.  We allocate 256 receive descriptors, each with a 2k
  * buffer (MCLBYTES), which gives us room for 50 jumbo packets.
  */
-#define        WM_NRXDESC              256
+#define        WM_NRXDESC              /* 256 */ 4096
 #define        WM_NRXDESC_MASK         (WM_NRXDESC - 1)
 #define        WM_NEXTRX(x)            (((x) + 1) & WM_NRXDESC_MASK)
 #define        WM_PREVRX(x)            (((x) - 1) & WM_NRXDESC_MASK)
@@ -3664,8 +3664,12 @@ wm_init(struct ifnet *ifp)
                CSR_WRITE(sc, WMREG_TDLEN, WM_TXDESCSIZE(sc));
                CSR_WRITE(sc, WMREG_TDH, 0);
                CSR_WRITE(sc, WMREG_TDT, 0);
+#if 0
                CSR_WRITE(sc, WMREG_TIDV, 375);         /* ITR / 4 */
                CSR_WRITE(sc, WMREG_TADV, 375);         /* should be same */
+#endif
+               CSR_WRITE(sc, WMREG_TIDV, 8);           /* linux magic */
+               CSR_WRITE(sc, WMREG_TADV, 32);          /* ditto */
 
                if ((sc->sc_flags & WM_F_NEWQUEUE) != 0)
                        CSR_WRITE(sc, WMREG_TXDCTL, TXDCTL_QUEUE_ENABLE
@@ -3722,8 +3726,12 @@ wm_init(struct ifnet *ifp)
                } else {
                        CSR_WRITE(sc, WMREG_RDH, 0);
                        CSR_WRITE(sc, WMREG_RDT, 0);
+#if 0
                        CSR_WRITE(sc, WMREG_RDTR, 375 | RDTR_FPD);      /* 
ITR/4 */
                        CSR_WRITE(sc, WMREG_RADV, 375);         /* MUST be same 
*/
+#endif
+                       CSR_WRITE(sc, WMREG_RDTR, 0 | RDTR_FPD);/* linux magic 
*/
+                       CSR_WRITE(sc, WMREG_RADV, 8);           /* ditto */
                }
        }
        for (i = 0; i < WM_NRXDESC; i++) {
@@ -3892,7 +3900,10 @@ wm_init(struct ifnet *ifp)
                  * divided by 4 to get "simple timer" behavior.
                  */
 
+#if 0
                sc->sc_itr = 1500;              /* 2604 ints/sec */
+#endif
+               sc->sc_itr = 1000000000 / (20000 * 256); /* linux defaults */
                CSR_WRITE(sc, WMREG_ITR, sc->sc_itr);
        }