Poor disk performance in NetBSD DomU, again

[Yi Qiao <qiaoy%qiaoy.info@localhost>]

Hello

  I own an Dell Zino HD, which I am trying to convert into a personal
cloud running XEN 4.2 + NetBSD Dom0. But after a week of
experimenting, I am stuck at the problem that disk io performance
seems to be dramatically reduced in NetBSD PV DomU.

  I've searched the web and the mail list. There seems to be quite a
few people having this problem, but none of the followings threads
resulted in a solution...

http://thread.gmane.org/gmane.os.netbsd.ports.xen/7806
http://thread.gmane.org/gmane.os.netbsd.ports.xen/7016

So here is my machine spec:
CPU: AMD Athlon II P360 (2.3G)
Memory: 8G
Harddrive:
  ahcisata0 port 0: device present, speed: 3.0Gb/s
  atabus0 at ahcisata0 channel 0
  wd0 at atabus0 drive 0
  wd0: drive supports 16-sector PIO transfers, LBA48 addressing
  wd0: 931 GB, 1938021 cyl, 16 head, 63 sec, 512 bytes/sect x 1953525168 sectors
  wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
  wd0(ahcisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6
(Ultra/133) (using DMA)


I installed a fresh copy of NetBSD-6.1.4 on wd0a (10G), and allocated
512M to wd0b as swap (since I'm planning on running Dom0 with 256M
memory). The rest of the hard drive went to wd0e, which was then made
into a lvm pv

Now here comes the interesting part. I tested hard drive read and
write throughput using dd in various situations, and the number I
report is average over four independent runs with 4k, 8k, 16k and 32k
block size. The commands I used to test write and read speed are

dd if=/dev/zero of=testfile bs=<bs> count=<4G/bs>
dd if=testfile of=/dev/null bs=<bs>

I tried to put conv=fdatasync at the end of the write test, no
significant difference was observed.

Here are the results:

NetBSD Native, wd0a: 111.93 MB/s (write); 113.73 MB/s (read)
NetBSD Native, lvm: 114.65 MB/s (write); 118.89 MB/s (read)
NetBSD Dom0, wd0a: 111.96 MB/s (write); 116.10 MB/s (read)
NetBSD Dom0, lvm: 115.45 MB/s (write); 122.30 MB/s (read)

NetBSD lvm based DomU: 16.11 MB/s (write); 116.78 MB/s (read)
NetBSD img file based DomU: 15.62 MB/s (write); 116.52 MB/s (read)

Ubuntu 14.04 lvm based PV DomU: 122.41 MB/s (write); 124.28 MB/s (read)
NetBSD lvm based DomU on Ubuntu Dom0: 8.65 MB/s (write); 118.72 MB/s (read)

A few observation from the data:
* The hard drive native speed is around 100 MB/s read / write, based
on the NetBSD native and NetBSD Dom0 tests. lvm does not incur any
penalty.
* NetBSD DomU has significant reduced write speed. Read speed remain
unaffected. I tried to mount the filesystem without log, or mount with
async. None of those helped
* Linux DomU does not seem to suffer from the write speed degradation.
* NetBSD DomU does not benefit from a Linux Dom0

I also tried to run NetBSD Dom0 directly from a usb drive, and
dedicated my entire hard drive to lvm. This did not help either.


So at this point I'm lost in terms of how to remedy the DomU write
speed loss, since there is little to tinker with in the config file.
I'm not ready to give NetBSD up just yet. I stopped running Linux
since kernel 2.6.32, and every time I'm forced to run it, I feel dirty
and obese. Any help would be greatly appreciated ;)


As a addendum, here is my NetBSD DomU config file. I'm using xl
toolchain, as per the recommendation of the xentools42 package. The
config file is adapted from
/usr/pkg/share/examples/xen/xlexample.pvlinux

NetBSD:
name = "netbsd"
#kernel = "/share/xen-pvkernels/nbsd/6.1.4/netbsd-INSTALL_XEN3_DOMU.gz"
kernel = "/share/xen-pvkernels/nbsd/6.1.4/netbsd-XEN3_DOMU.gz"
memory = 512
vcpus = 1
vif = [ 'bridge=bridge0' ]
disk = [ 'phy:/dev/mapper/vg0-netbsd,xvda,rw' ]

Linux:
name = "ubuntu"
# following three lines are for booting the net-install kernel
# kernel = "/share/xen-pvkernels/ubuntu/trusty/vmlinuz"
# ramdisk = "/share/xen-pvkernels/ubuntu/trusty/initrd.gz"
# extra = "debian-installer/exit/always_halt=true -- console=hvc0"
bootloader = "pygrub"
memory = 512
vcpus = 1
vif = [ 'bridge=bridge0' ]
disk = [ 'phy:/dev/mapper/vg0-ubuntu,xvda,w' ]