Re: Interrupt time inflation on Xen

[Christoph Badura <bad%bsd.de@localhost>]

On Sun, Mar 13, 2016 at 07:08:04PM -0400, Thor Lancelot Simon wrote:
> I had occasion a couple of days ago to try to block-copy a very large
> filesystem from a xen dom0 to another machine across a fast local network.
> [...]
> Shockingly, I saw the system go to 0% idle time, with 45-55% "Interrupt"
> and the rest "System".  Interrupts per second were a comparatively low 1500,
> about 500 disk and 1000 network.  Throughput was horrible -- about 35MB/sec.

Maybe this is not specific to interrupts.  E.g. doing a build of
xentools45 in a dom0 shows inflated "System" time.  Even when there is
little disk I/O (50-200 IO/s and systat showing 0% "Interrupt" time),
"System" time is in the 30%-45% range.  Doing the same "bare metal" on the
same box uses way less "System" time.  It mostly hovers around 20% when
booted non-SMP (i.e. with boot flag -1), depending on how many forks are
done.

Maybe there are some very expensive hypervisor operations being called
that lead to all "System" and "Interrupt" time.  Perhaps waiting for the
hypervisor to complete the operations.

Just dd'ing from disk to /dev/null doesn't show increased "Interrupt" time
under xen.  (dd if=/dev/rwd0d of=/dev/null bs=64k count=10000)
It's about 16% "System" time while the dd runs, otherwise idle.  I get
about 125 MB/s throughput.

This is the same with Xen4.2 and Xen4.5.

The machine is a Dell Precision T1600 w/16GB RAM (8GB in the dom0).
NetBSD-7 from the daily releng builds: XEN3_DOM0.201603122210Z.

vcpu0 at hypervisor0: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz, id 0x206a7

wd0 is a Kingston SV300 SSD on ahci at 6 Gb/s link speed.

Disabling VT-D makes no difference.

--chris