Re: Poor disk performance in NetBSD DomU, again

[Yi Qiao <qiaoy%qiaoy.info@localhost>]

Hi Manuel

  Sorry about that. It's all PV guests. And I don't think partition
alignments is the dominating issue here since after each experiment, I
remove the lvm volume for that guest, and create a new one for the
next guest.

  I also did some more experiment with HVM guests, and here are the
results (all with NetBSD Dom0)

  NetBSD: 61.66 MB/s (write); 73.69 MB/s (read)
  FreeBSD-9.2: 98.61 MB/s (write); 103.81 MB/s (read)
  OpenBSD-5.5: 24.91 MB/s (write); 59.96 MB/s (read)
  Ubuntu 14.04: 129 MB/s (write); 131 MB/s (read)

It's interesting to see that NetBSD HVM DomU actually out-performed PV
DomU, but still lagging behind FreeBSD and Ubuntu.

All guest was configured to have 512M memory and 1 vcpu.



On Tue, May 13, 2014 at 6:39 AM, Manuel Bouyer 
<bouyer%antioche.eu.org@localhost> wrote:
> On Sun, May 11, 2014 at 05:24:32PM -0400, Yi Qiao wrote:
>> Hello
>>
>>   I own an Dell Zino HD, which I am trying to convert into a personal
>> cloud running XEN 4.2 + NetBSD Dom0. But after a week of
>> experimenting, I am stuck at the problem that disk io performance
>> seems to be dramatically reduced in NetBSD PV DomU.
>>
>>   I've searched the web and the mail list. There seems to be quite a
>> few people having this problem, but none of the followings threads
>> resulted in a solution...
>>
>> http://thread.gmane.org/gmane.os.netbsd.ports.xen/7806
>> http://thread.gmane.org/gmane.os.netbsd.ports.xen/7016
>>
>> So here is my machine spec:
>> CPU: AMD Athlon II P360 (2.3G)
>> Memory: 8G
>> Harddrive:
>>   ahcisata0 port 0: device present, speed: 3.0Gb/s
>>   atabus0 at ahcisata0 channel 0
>>   wd0 at atabus0 drive 0
>>   wd0: drive supports 16-sector PIO transfers, LBA48 addressing
>>   wd0: 931 GB, 1938021 cyl, 16 head, 63 sec, 512 bytes/sect x 1953525168 
>> sectors
>>   wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 6 (Ultra/133)
>>   wd0(ahcisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 6
>> (Ultra/133) (using DMA)
>>
>>
>> I installed a fresh copy of NetBSD-6.1.4 on wd0a (10G), and allocated
>> 512M to wd0b as swap (since I'm planning on running Dom0 with 256M
>> memory). The rest of the hard drive went to wd0e, which was then made
>> into a lvm pv
>>
>> Now here comes the interesting part. I tested hard drive read and
>> write throughput using dd in various situations, and the number I
>> report is average over four independent runs with 4k, 8k, 16k and 32k
>> block size. The commands I used to test write and read speed are
>>
>> dd if=/dev/zero of=testfile bs=<bs> count=<4G/bs>
>> dd if=testfile of=/dev/null bs=<bs>
>>
>> I tried to put conv=fdatasync at the end of the write test, no
>> significant difference was observed.
>>
>> Here are the results:
>>
>> NetBSD Native, wd0a: 111.93 MB/s (write); 113.73 MB/s (read)
>> NetBSD Native, lvm: 114.65 MB/s (write); 118.89 MB/s (read)
>> NetBSD Dom0, wd0a: 111.96 MB/s (write); 116.10 MB/s (read)
>> NetBSD Dom0, lvm: 115.45 MB/s (write); 122.30 MB/s (read)
>>
>> NetBSD lvm based DomU: 16.11 MB/s (write); 116.78 MB/s (read)
>> NetBSD img file based DomU: 15.62 MB/s (write); 116.52 MB/s (read)
>>
>> Ubuntu 14.04 lvm based PV DomU: 122.41 MB/s (write); 124.28 MB/s (read)
>> NetBSD lvm based DomU on Ubuntu Dom0: 8.65 MB/s (write); 118.72 MB/s (read)
>>
>> A few observation from the data:
>> * The hard drive native speed is around 100 MB/s read / write, based
>> on the NetBSD native and NetBSD Dom0 tests. lvm does not incur any
>> penalty.
>> * NetBSD DomU has significant reduced write speed. Read speed remain
>> unaffected. I tried to mount the filesystem without log, or mount with
>> async. None of those helped
>> * Linux DomU does not seem to suffer from the write speed degradation.
>> * NetBSD DomU does not benefit from a Linux Dom0
>>
>> I also tried to run NetBSD Dom0 directly from a usb drive, and
>> dedicated my entire hard drive to lvm. This did not help either.
>>
>>
>> So at this point I'm lost in terms of how to remedy the DomU write
>> speed loss, since there is little to tinker with in the config file.
>> I'm not ready to give NetBSD up just yet. I stopped running Linux
>> since kernel 2.6.32, and every time I'm forced to run it, I feel dirty
>> and obese. Any help would be greatly appreciated ;)
>
> You're not saying if you're using PV or HVM domUs.
> Also, partition alignements can make a difference.
>
> --
> Manuel Bouyer <bouyer%antioche.eu.org@localhost>
>      NetBSD: 26 ans d'experience feront toujours la difference
> --