Re: Interrupt time inflation on Xen

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

Here's a few tests I ran.
Hardware is a Dell optiplex 755 (core 2 duo, WDC WD20EARS-00MVWB0, on Intel
AHCI):

Xen45/netbsd-6:
borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes transferred in 53.957 secs (121459680 bytes/sec)
systat says:
  11.4% Sy   0.8% Us   0.0% Ni   2.4% In  85.4% Id
1850 ops/s, 1850 irq/s on ioapic0 pin 18

netbsd-6 bare metal, booted with -1 (as XEN3_DOM0 is not SMP yet):
borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes transferred in 53.882 secs (121628744 bytes/sec)
systat says:
   2.6% Sy   0.0% Us   0.0% Ni   1.8% In  95.6% Id
about 1850 ops/s, 1850 irq/s on ioapic0 pin 18

xen45/netbsd-7:
borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes transferred in 54.258 secs (120785874 bytes/sec)
systat says:
  15.8% Sy   0.8% Us   0.0% Ni   0.6% In  82.8% Id    pages
about 1850 ops/s, 1850 irq/s on ioapic0 pin 18

netbsd-7 bare metal, booted with -1 (as XEN3_DOM0 is not SMP yet):
borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
6553600000 bytes transferred in 54.175 secs (120970927 bytes/sec)
systat says:
   3.6% Sy   0.2% Us   0.0% Ni   1.4% In  94.8% Id
about 1850 ops/s, 1850 irq/s on ioapic0 pin 18

xen31/netbsd-7:
borneo:/home/bouyer#dd if=/dev/rwd0d of=/dev/null bs=64k count=100000
100000+0 records in
100000+0 records out
systat says:
  15.6% Sy   0.2% Us   0.0% Ni   0.8% In  83.4% Id
about 1850 ops/s, 1850 irq/s on ioapic0 pin 18

No, I'm not seeing twice the interrupts as you see with the SCSI controllers.
I suspect it could be interrupt coalescing, which may kick in with a bare-metal
kernel and not with the Xen one, as the Xen one is slower.

For I/Os, Xen is not slower in my case but uses much more system time.
This is probably because on this hardware, the limiting factor is the
hard disk and not the CPU.
This time is not spent in hard interrupt handlers, I actually think it's
MMU-related (page remapping or invalidations). It makes sense, as in the
Xen/PV case these operations needs an hypercall. There's no big differences
between netbsd-6 and netbsd-7, or Xen versions.

I re-ran tests with the attached program, which should avoid reading data
from disk (the data should be in the disks's cache, so we're benchmarking
the SATA link/controller, and the data move between kernel and userland).

On Xen/netbsd-7:
  23.4% Sy   0.4% Us   0.0% Ni   0.8% In  75.4% Id
borneo:/home/bouyer#./tst /dev/rwd0d 100000
37753400 us, 165.548004 MB/s
UP bare-metal:
   4.0% Sy   0.0% Us   0.0% Ni   3.6% In  92.4% Id
borneo:/home/bouyer#./tst /dev/rwd0d 100000
30945314 us, 201.969190 MB/s

Here we have a true difference (also, Xen/netbsd-7 can't keep the disk more
than 70% busy, while bare metal is more than 99% - maybe because interrupt
latency is higher with Xen). The result is the same with different Xen 
versions.

Maybe some improvement is possible in pmap.c, this needs to be looked at.
But I don't think we'll get more than a few %.

Maybe also a XEN3PAE_DOM0 would do better (with amd64/xen the kernel
runs unprivileged and has a completely separate vm space from
userland, while i386PAE kernel runs in ring 1 and shares its vm space with
userland - this makes a big difference in pmap). I never tried, but it should
be possible to run a amd64 Xen with a i386pae dom0.

The real fix would be to run the dom0 kernel as a HVM domain on hardware
that supports it. But this requires quite a bit of work.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference

#include <fcntl.h>
#include <unistd.h>
#include <sys/time.h>

main(int argc, char **argv)
{
	static char buf[64*1024];
	int fd, i;
	struct timeval tv0, tv1;
	int t;

	fd = open(argv[1], O_RDONLY, 0);
	if (fd < 0) {
		perror("open");
		exit(1);
	}
	if (gettimeofday(&tv0, NULL) < 0) {
		perror("gettimeofday");
		exit(1);
	}
	for (i = 0; i < atoi(argv[2]); i++) {
		if (read(fd, buf, sizeof(buf)) != sizeof(buf)) {
			perror("read");
			exit(1);
		}
		if (lseek(fd, 0, SEEK_SET) < 0) {
			perror("seek");
			exit(1);
		}
			
	}
	if (gettimeofday(&tv1, NULL) < 0) {
		perror("gettimeofday");
		exit(1);
	}
	t = (tv1.tv_sec - tv0.tv_sec) * 1000000;
	t = t + tv1.tv_usec - tv0.tv_usec;
	printf("%d us, %f MB/s\n", t,
	    ((double)64 * (double)i / 1024) / ((double)t / 1000000));
	exit(0);
}