Subject: Re: Why is my rmt so slow or what takes exactly 0.2 seconds?
To: Manuel Bouyer <bouyer@antioche.eu.org>
From: Bill Studenmund <wrstuden@netbsd.org>
List: tech-net
Date: 08/25/2005 11:50:52
--XOIedfhf+7KOe/yw
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Aug 17, 2005 at 06:22:34PM +0200, Manuel Bouyer wrote:
> On Wed, Aug 17, 2005 at 06:13:03PM +0200, Edgar Fu? wrote:
> > I'm not entirely sure my problem is net related.
> >=20
> > The problem occurs on 2.0.2/i386 in case it matters.
> >=20
> > I wondered why my dump via (ssh tunneled) rmt was so slow (80kB/s).
> >=20
> > I boiled it down to having both dump and rmt run on the same machine
> > (so it can't be the network -- switches or something) and outputting
> > to /dev/null (so it can't be the tape device). Still only 80kB/s.
> >=20
> > Dumping directly to /dev/nrst0 gives close to 2MB/s, so it's not the di=
sk.
> >=20
> > I ktraced both ends (dump/ssh and ssh/rmt) and inspected the kdumps
> > with time stamps.
> >=20
> > It looks like, from time to time, the rmt process issues a read call
> > that only returns after nearly exactly 0.2 seconds for no apparent reas=
on.
> > Sometimes, this happens while reading large chunks of tape data, but so=
metimes
> > it even happens reading a single command character of the rmt protocol.
> >=20
> > Any ideas what keeps the rmt process scheduled away for 0.2 seconds?
>=20
> It could be a tcp issue, waiting for more data before sending a packet,
> while the dump/rmt protocol is waiting for a reply. Try to play with the
> various tcp sysctls.

I assume this is over the loopback interface? If so, you might try
lowering the MTU on the loopback interface. I have an application that
tries to shove a lot of data over loopback, and if the transmissions get
too one-sided, throughput plummets. As in 256k transactions transfering a
given size happen at 43 MB/sec and 512k transactions (transfering same net=
=20
size) are at like 0.8 MB/sec. Yes, that's almost two orders of magnitude=20
drop, just from increasing the transfer size.

=46rom looking at the server's behavior, the client just seems to be pausin=
g=20
in sending the data. As best I can tell, it's tcp that's pausing. And this=
=20
is with 256k send and receive buffers on the server (which is receiving=20
these transactions).

If I drop the loopback mtu from 33192 to say 18000, the issue goes away.

Take care,

Bill

--XOIedfhf+7KOe/yw
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (NetBSD)

iD8DBQFDDhMMWz+3JHUci9cRAoWUAJ4qjgTnUDg49QZ2fRVwyTzsygBbFACfQby/
WgFVe9PjN1QfJ2RcpqDsH9Y=
=baoC
-----END PGP SIGNATURE-----

--XOIedfhf+7KOe/yw--