Interrupt time inflation on Xen

[Thor Lancelot Simon <tls%panix.com@localhost>]

I had occasion a couple of days ago to try to block-copy a very large
filesystem from a xen dom0 to another machine across a fast local network.

I tried this:

sysctl -w kern.sbmax=10000000
sysctl -w net.inet.tcp.sendbuf_auto=0
dd if=/dev/rsd0g bs=1048576 | ttcp -t -b 2097152 -l 131072 -fm -p 9001 172.0.0.1

On the far end, I had simply the corresponding ttcp -r job (with corresponding
socket/tcp sysctls in place) redirected to a file.

The underlying hardware is a RAID5 of 300GB SAS drives on an Areca 1680.
The Ethernet controller in use is a 'wm'.  All supported offload functionality
is turned on (turning it off just slows things down).

Shockingly, I saw the system go to 0% idle time, with 45-55% "Interrupt"
and the rest "System".  Interrupts per second were a comparatively low 1500,
about 500 disk and 1000 network.  Throughput was horrible -- about 35MB/sec.

I rebooted the system to a GENERIC kernel; throughput was 100MB/sec and
peaked at 200MB/sec when I piped the data through gzip -1.  The system
was mostly idle with under 2% "Interrupt" time.

Interrupts/sec were significantly higher due to both the higher data rate
and about 800 TLB shootdown interrupts/sec (the GENERIC kernel runs MP
on this system).

Kernels are both netbsd-7 branch from just around the time of the release,
XEN3_DOM0 and GENERIC.  Neither kernel has DEBUG nor DIAGNOSTIC nor KMEMSTATS.

Do we have some kind of terrible problem that causes a 20% increase in
interrupt time when running under Xen?  Note the Areca driver (arcmsr) is
a SCSI driver so it is giant-locked, but the XEN dom0 is UP, so this is
probably a red herring.

I am going to install a different disk controller and see what happens,
but I am not totally optimistic.  Does anyone have any idea what might be
going on?

-- 
  Thor Lancelot Simon	                                     tls%panix.com@localhost

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."      - H.L.A. Hart