Re: NetBSD 5.1 TCP performance issue (lots of ACK)

[Greg Oster <oster%cs.usask.ca@localhost>]

On Fri, 28 Oct 2011 16:10:36 +0200
Manuel Bouyer <bouyer%antioche.eu.org@localhost> wrote:

> On Thu, Oct 27, 2011 at 07:51:43PM +0200, Manuel Bouyer wrote:
> > On Thu, Oct 27, 2011 at 12:00:33PM -0400, Thor Lancelot Simon wrote:
> > > It's possible this has to do with the interrupt moderation
> > > tuning.  I believe we've been pending the checkin of better
> > > values than the ones I worked out from the documentation for
> > > quite some time -- there were highly unobvious performance
> > > effects with small buffers.  Simon did a bunch of testing and
> > > concluded, as I recall, that the values used by Intel in the
> > > Linux driver were "magic" and that we should use those, not mine.
> > > 
> > > If this hasn't been adjusted to match the Linux driver, you might
> > > want to take a quick look at the values it uses and see whether
> > > they yield better small-buffer performance in your case.
> > 
> > I looked quickly at this and came up with the attached patch.
> > 
> > With this (installed on both NetBSD hosts) I get mittiged results:
> > - the NetBSD client against the linux server gets degranded and
> > unstable performances several runs gives large variations in speed
> > - the NetBSD client against the NetBSD server gets better
> > performances in average (but still not in the 90MB range) and also
> > with large variations between runs
> > - the linux client against the NetBSD server gets a little boost and
> >   the speed is stll stable between runs
> > - ttcp performances between NetBSD hosts gets a little boost too,
> >   and the speed is stll stable between runs
> > 
> > But I do get Ierrs on both NetBSD hosts now, with the ttcp or
> > glusterfs test. I don't know where these errors comes from. Linux
> > has no errors. I don't think it's wm_add_rxbuf(), netstat -m and
> > vmstat -m shows no issues with mbuf allocations.
> > So I guess these are errors at the adapter level, we may need to
> > change more things to match these values.
> > Also, linux seems to be using more advanced features for these
> > adapters, this is something we may have to look at too.
> 
> Here is an updated patch. The key point to avoid the receive errors is
> to do another BUS_DMASYNC after reading wrx_status, before reading the
> other values to avoid reading e.g. len before status gets updated.
> The errors were because of 0-len receive descriptors.
> 
> With this I get 113MB/s with the ttcp test, and between 70 and 90MB/s
> with glusterfs. the NetBSD client now gets the same speed with a
> NetBSD or linux server.
> 
> In the patch there is changes for the WM_F_NEWQUEUE adapters but they
> may not be correct. When using WM_F_NEWQUEUE for the i80003 (which
> the linux driver does), performances are a little lower and I get
> a high interrupt rate - just as if interrupt mitigation was not
> working.

Hi Manuel

Have there been issues with these patches that prevent them from being
applied to -current and/or pulled up?  (says he who saw reasonably
abysmal network performance this weekend on a bunch of machines with
wm0 in use, and would like to see some better performance.)

Thanks.

Later...

Greg Oster