I've been playing with some iSCSI code, and noticed a very odd behavior
with a userland test program. It performs a series of iSCSI pings (NOP-OUT
with optional NOP-IN echo). They all fly along (less than like .2 seconds
for 1000 itterations), until the one test case where the iSCSI target is
echoing 4k of data back. That test takes 200 seconds.

I looked into it, and the problem is that, for that one test case, the
target is sending data back in two writes. The first is a 48-byte iSCSI
PDU, the other is the 4k of test data. For some reason, the target waits
for an ack from the initiator before sending the 4k response. That ack
takes .199 seconds to arrive, thus adding the delay.

What I really don't get is that both sides are doing the same write
sequence (48 bytes, 4k), with the same tcp options (TCP_NODELAY), running
on the same machine (using localhost), but only one side of it is having
to delay.

From looking at the packets in Ethereal, it looks like the tcp window is
around 12k or 16k, so I don't see why a 4k write blocks.

Jason suggested I try setting 'sysctl -w net.inet.tcp.ack_on_push=1'. That
got rid of the delay.

I've created tcpdumps of sessions with and without the delay. They are at:

With delay:
ftp://ftp.netbsd.org/pub/NetBSD/misc/wrstuden/tcpdumps/smalldump

W/o delay (ack_on_push = 1):
ftp://ftp.netbsd.org/pub/NetBSD/misc/wrstuden/tcpdumps/smalldump1

In each dump, I've included some packets echoing data quickly (1k of data
I think), and packets in the case which has the problem.

Thoughts?

Take care,

Bill