Re: xbd and large sectors

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Fri, Jul 21, 2023 at 01:37:27AM +0200, Christian Kujau wrote:
> On Thu, 20 Jul 2023, Greg Troxel wrote:
> > Over on netbsd-users@, we have been discussing a problem where a dom0
> > has a disk with 4K sectors and it is being provided to a domU.  Things
> 
> Thank you Greg for taking this to port-xen! For everyone else, this is my 
> initial post: https://mail-index.netbsd.org/netbsd-users/2023/07/20/msg029875.html
> 
> Some more details were requested over there. So, for example when booting 
> NetBSD 9.3 in Xen HVM mode, the disk is represented as wd(4) instead and 
> the disk size (8 GB) is recognized correctly:
> 
>  wd0 at atabus0 drive 0
>  wd0: <QEMU HARDDISK>
>  wd0: drive supports 16-sector PIO transfers, LBA48 addressing
>  wd0: 8192 MB, 16644 cyl, 16 head, 63 sec, 512 bytes/sect x 16777216 sectors
>  wd0: 32-bit data port
>  wd0: drive supports PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100)
>  wd0(piixide0:0:0): using PIO mode 4, DMA mode 2 (using DMA)

this is a 512-bytes sector drive too, not 4K

> 
> Curiously enough, when booting boot.iso from NetBSD-daily (202307191930Z) 
> in HVM mode again, the disk appears to be accessed via xbd(4), and shows 
> an incorrect disk size, and errors too:
> 
>  xbd0 at xenbusO id 51712
>  xbd0: using event channel 13
>  xbd0: 65536 MB, 512 bytes/sect x 134217726 sectors
>  xbd0: backend features 0xc<PERSISTENT, INDIRECT>
>  unknown type vkbdxbd0d: error reading fsbn 0 (xbd0 bn 0; cn 0 tn 0 sn 0)
>    at xenbus0xbd0: dos partition I/0 error
>    id 0xbd0d: error reading fsbn 0 of 0-31 (xbd0 bn 0; ch 0 tn 0 sn 0)
>  xbd0d: error reading fsbn 1 (xbd0 bn 1; cn 0 tn 0 sn 1)
>  xbd0d: error reading fsbn 0 (xbd0 bn 0; cn 0 tn 0 sn 0)
>  xbd0: dos partition I/0 error
> 
> Let's find out if I'm allowed to attach files here: I've attached the 
> output of "xenstore-ls" on the Dom0 domain, the running NetBSD domain 
> (still in HVM mode) has id "35", with three disks attached:
> 
>  disk = [ 'format=raw, vdev=xvda, access=rw, target=/dev/vg0/netbsd-disk0',
>           'format=raw, vdev=xvdb, access=rw, target=/dev/vg0/netbsd-disk1',
>           'format=raw, vdev=xvdc, access=ro, devtype=cdrom, target=/var/local/vm/boot/netbsd/netbsd-boot.iso.202307191930Z',
> 
> I hope this helps with debugging somehow.

So the relevant part is:

     35 = ""
      51712 = ""
       frontend = "/local/domain/35/device/vbd/51712"
       params = "/dev/vg0/netbsd-disk0"
       script = "/etc/xen/scripts/block"
       frontend-id = "35"
       online = "1"
       removable = "0"
       bootable = "1"
       state = "4"
       dev = "xvda"
       type = "phy"
       mode = "w"
       device-type = "disk"
       discard-enable = "1"
       specification = "xen"
       feature-max-indirect-segments = "256"
       multi-queue-max-queues = "4"
       max-ring-page-order = "4"
       physical-device = "fd:5"
       physical-device-path = "/dev/dm-5"
       hotplug-status = "connected"
       feature-flush-cache = "1"
       feature-discard = "0"
       feature-barrier = "1"
       feature-persistent = "1"
       sectors = "16777216"
       info = "0"
       sector-size = "4096"
       physical-sector-size = "4096"
      51728 = ""
       frontend = "/local/domain/35/device/vbd/51728"
       params = "/dev/vg0/netbsd-disk1"
       script = "/etc/xen/scripts/block"
       frontend-id = "35"
       online = "1"
       removable = "0"
       bootable = "1"
       state = "4"
       dev = "xvdb"
       type = "phy"
       mode = "w"
       device-type = "disk"
       discard-enable = "1"
       specification = "xen"
       feature-max-indirect-segments = "256"
       multi-queue-max-queues = "4"
       max-ring-page-order = "4"
       physical-device = "fd:6"
       physical-device-path = "/dev/dm-6"
       hotplug-status = "connected"
       feature-flush-cache = "1"
       feature-discard = "0"
       feature-barrier = "1"
       feature-persistent = "1"
       sectors = "33554432"
       info = "0"
       sector-size = "4096"
       physical-sector-size = "4096"

the backend reports, for xbd0, 16777216 sectors, which we compute as
16777216 * 4k = 64G (and the same 8x larger for xbd1). So the number
of sectors reported here are 512-bytes sectors, not 4k.

Reading the linux sources, the xbd driver pass "sectors" directly to
set_capacity(), which expects the disk size in 512-byte sectors, so
the backend is doing what linux expects, and NetBSD is miscomputing
the disk size when doing sectors * sector-size.

A NetBSD dom0 will never report 4k sectors to the guest so we can just fix
the frontend, and there's no backward compat issue.

I will commit a fix to HEAD later today.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--