Re: TSO not working?

[David Young <dyoung%pobox.com@localhost>]

On Sat, Oct 29, 2011 at 10:25:54PM +0200, Manuel Bouyer wrote:
> On Fri, Oct 28, 2011 at 02:24:04PM -0400, Thor Lancelot Simon wrote:
> > > I added a sc_ev_txlargetso, which gets incremented by:
> > >   if (m0->m_pkthdr.len > 32768)
> > >           WM_EVCNT_INCR(&sc->sc_ev_txlargetso);
> > 
> > It'd be interesting, perhaps, to see how often we were actually even > 1500.
> 
> I added this:
>                 if (m0->m_pkthdr.len >= 32768)
>                         WM_EVCNT_INCR(&sc->sc_ev_txtsolarge);
>                 if (m0->m_pkthdr.len >= 16384)
>                         WM_EVCNT_INCR(&sc->sc_ev_txtsomedium);
>                 if (m0->m_pkthdr.len >= 1500)
>                         WM_EVCNT_INCR(&sc->sc_ev_txtsomtu);
> 
> and here's the results:
> for ttcp against a NetBSD client:
> wm0 txtso                                    188555      695 misc
> wm0 txtsomtu                                 188555      695 misc
> wm0 txtsomedium                                  92        0 misc
> wm0 txtsolarge                                    4        0 misc
> 
> glusterfs against a NetBSD client:
> wm0 txtso                                    114607      690 misc
> wm0 txtsomtu                                 114607      690 misc
> wm0 txtsomedium                                1496        9 misc
> wm0 txtsolarge                                  709        4 misc
> 
> glusterfs against a linux client:
> wm0 txtso                                     86567     1185 misc
> wm0 txtsomtu                                  86567     1185 misc
> wm0 txtsomedium                                6897       94 misc
> wm0 txtsolarge                                 2391       32 misc
> 
> > 
> > > ttcp -t -l65536 -b524288 -D xen1-priv < /glpool/truc
> > > 
> > > (64k writes but a 512k TCP buffer) I get no "largetso".
> > 
> > Does the window ever have 32K of space, I wonder?  Maybe the delay is
> > not large enough to ever let it open that wide.
> 
> We should have this in the trace ?

Should be.

Does anything keep NetBSD from dribbling MTU-size segments onto the
interface output queue while one or two super-segments (size > MTU)
still wait to be transmitted?

The TSO code definitely tries to prevent pathological alternation
between super-segments and runts[1], however, it looks to me like
nothing prevents alternation of super-segments and MTU-size segments.
That is not pathological, perhaps, but we can probably do better.

Dave

[1] This code is in tcp_output():

        if (len > txsegsize) {
                if (use_tso) {
                        /*
                         * Truncate TSO transfers to IP_MAXPACKET, and make
                         * sure that we send equal size transfers down the
                         * stack (rather than big-small-big-small-...).
                         */
#ifdef INET6
                        CTASSERT(IPV6_MAXPACKET == IP_MAXPACKET);
#endif
                        len = (min(len, IP_MAXPACKET) / txsegsize) * txsegsize;
                        if (len <= txsegsize) {
                                use_tso = 0;
                        }
                } else
                        len = txsegsize;

    It looks to me like the code prevents an alternation large,
    sub-t_segsz, large, sub-t_segsz, large, but not an alternation
    large, t_segsz, large, t_segsz, large....

-- 
David Young             OJC Technologies is now Pixo
dyoung%pixotech.com@localhost     Urbana, IL   (217) 344-0444 x24