I had occasion a couple of days ago to try to block-copy a very large
filesystem from a xen dom0 to another machine across a fast local network.
I tried this:
sysctl -w kern.sbmax=10000000
sysctl -w net.inet.tcp.sendbuf_auto=0
dd if=/dev/rsd0g bs=1048576 | ttcp -t -b 2097152 -l 131072 -fm -p 9001 172.0.0.1
On the far end, I had simply the corresponding ttcp -r job (with corresponding
socket/tcp sysctls in place) redirected to a file.
The underlying hardware is a RAID5 of 300GB SAS drives on an Areca 1680.
The Ethernet controller in use is a 'wm'. All supported offload functionality
is turned on (turning it off just slows things down).
Shockingly, I saw the system go to 0% idle time, with 45-55% "Interrupt"
and the rest "System". Interrupts per second were a comparatively low 1500,
about 500 disk and 1000 network. Throughput was horrible -- about 35MB/sec.