On Fri, Jul 12, 2002 at 12:55:33PM -0700, Bill Studenmund wrote:

 > At least for what I'm seeing, that sounds stupid. I have a 16k window. I
 > send 48 bytes. I then can't send 4096 bytes until the 48 are
 > acknowledged?? I could see that if I wanted to write 16k, I have to wait
 > (and I have a stupid window size).

This is the way TCP slow start works.  Conventional wisdom is that
it's the packet count, not the byte count, that matters when dealing
with congestion issues.  Sure, the receiver has that much room, but
you have to avoid congesting the all the stuff in-between.

For a router, a 128-byte packet is just as expensive as a 1500-byte
packet -- they each consume 1 queue slot and the same amount of packet
buffer space (since the network interface driver pre-allocates buffers
for maximum-sized packets ahead of time, and the data spends so little
time in the router that moving it to a smaller packet buffer isn't worth
the effort).

 > But with a 16k window of which 48 bytes are filled, why can't I send 4k?
 > Sounds like a bug somewhere...

The window the receiver is advertising back to you (your SND_WND) is 16k,
but your congestion window hasn't opened up yet.

The way this works is that your CWND starts out at 1 packet (the Initial
Window).  For each ACK you get back, it opens up 2 more packets.  Eventually,
your CWND opens up to match your SND_WND.

Now, even though things are counted as packets, the BSD TCP still likes
to think about bytes.  So, the way available CWND is computed is:

	initial-window * tx-segsize

If you are NOT using PMTU, then tx-segsize is tcp_mssdflt, which defaults
to 512.  If you are using PMTU, then it's the MTU of the outgoing interface,
adjusted for headers, etc.

It might be worth playing with the net.inet.tcp.init_win parameter -- crank
it up to 4, or something, and see if that makes any difference.  Also make
sure you have PMTU enabled (net.inet.ip.mtudisc).

If you know you don't have to go through a gateway, then it might be
reasonable to increase the initial CWND (i.e. have one for "local"
vs. "foreign" nets).  I believe FreeBSD has this, and it might be
worth experimenting with.  I will try and cook up a patch for it this
weekend or early next week.

-- 
        -- Jason R. Thorpe <thorpej@wasabisystems.com>