Re: Poor disk performance in NetBSD DomU, again

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Sun, May 11, 2014 at 05:24:32PM -0400, Yi Qiao wrote:
> Hello
> 
>   I own an Dell Zino HD, which I am trying to convert into a personal
> cloud running XEN 4.2 + NetBSD Dom0. But after a week of
> experimenting, I am stuck at the problem that disk io performance
> seems to be dramatically reduced in NetBSD PV DomU.
> 
>   I've searched the web and the mail list. There seems to be quite a
> few people having this problem, but none of the followings threads
> resulted in a solution...
> 
> http://thread.gmane.org/gmane.os.netbsd.ports.xen/7806
> http://thread.gmane.org/gmane.os.netbsd.ports.xen/7016
> 
> So here is my machine spec:
> CPU: AMD Athlon II P360 (2.3G)
> Memory: 8G
> Harddrive:
>   ahcisata0 port 0: device present, speed: 3.0Gb/s
>   atabus0 at ahcisata0 channel 0
>   wd0 at atabus0 drive 0
>   wd0: drive supports 16-sector PIO transfers, LBA48 addressing
>   wd0: 931 GB, 1938021 cyl, 16 head, 63 sec, 512 bytes/sect x 1953525168 
> sectors
>   wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
>   wd0(ahcisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6
> (Ultra/133) (using DMA)
> 
> 
> I installed a fresh copy of NetBSD-6.1.4 on wd0a (10G), and allocated
> 512M to wd0b as swap (since I'm planning on running Dom0 with 256M
> memory). The rest of the hard drive went to wd0e, which was then made
> into a lvm pv
> 
> Now here comes the interesting part. I tested hard drive read and
> write throughput using dd in various situations, and the number I
> report is average over four independent runs with 4k, 8k, 16k and 32k
> block size. The commands I used to test write and read speed are
> 
> dd if=/dev/zero of=testfile bs=<bs> count=<4G/bs>
> dd if=testfile of=/dev/null bs=<bs>
> 
> I tried to put conv=fdatasync at the end of the write test, no
> significant difference was observed.
> 
> Here are the results:
> 
> NetBSD Native, wd0a: 111.93 MB/s (write); 113.73 MB/s (read)
> NetBSD Native, lvm: 114.65 MB/s (write); 118.89 MB/s (read)
> NetBSD Dom0, wd0a: 111.96 MB/s (write); 116.10 MB/s (read)
> NetBSD Dom0, lvm: 115.45 MB/s (write); 122.30 MB/s (read)
> 
> NetBSD lvm based DomU: 16.11 MB/s (write); 116.78 MB/s (read)
> NetBSD img file based DomU: 15.62 MB/s (write); 116.52 MB/s (read)
> 
> Ubuntu 14.04 lvm based PV DomU: 122.41 MB/s (write); 124.28 MB/s (read)
> NetBSD lvm based DomU on Ubuntu Dom0: 8.65 MB/s (write); 118.72 MB/s (read)
> 
> A few observation from the data:
> * The hard drive native speed is around 100 MB/s read / write, based
> on the NetBSD native and NetBSD Dom0 tests. lvm does not incur any
> penalty.
> * NetBSD DomU has significant reduced write speed. Read speed remain
> unaffected. I tried to mount the filesystem without log, or mount with
> async. None of those helped
> * Linux DomU does not seem to suffer from the write speed degradation.
> * NetBSD DomU does not benefit from a Linux Dom0
> 
> I also tried to run NetBSD Dom0 directly from a usb drive, and
> dedicated my entire hard drive to lvm. This did not help either.
> 
> 
> So at this point I'm lost in terms of how to remedy the DomU write
> speed loss, since there is little to tinker with in the config file.
> I'm not ready to give NetBSD up just yet. I stopped running Linux
> since kernel 2.6.32, and every time I'm forced to run it, I feel dirty
> and obese. Any help would be greatly appreciated ;)

You're not saying if you're using PV or HVM domUs. 
Also, partition alignements can make a difference.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--