Fwd: More ZFS testing in 9.0_BETA

[Chavdar Ivanov <ci4ic4%gmail.com@localhost>]

---------- Forwarded message ---------
From: Chavdar Ivanov <ci4ic4%gmail.com@localhost>
Date: Sat, 24 Aug 2019 at 23:54
Subject: Re: More ZFS testing in 9.0_BETA
To: Jason Mitchell <jmitchel%bigjar.com@localhost>


gpt can see properly zvols:
...
# gpt show /dev/zvol/dsk/pail/iscsia
     start      size  index  contents
         0         1         PMBR
         1         1         Pri GPT header
         2        32         Pri GPT table
        34     32734      1  GPT part - Windows reserved
     32768  62877696      2  GPT part - Windows basic data
  62910464      4063         Unused
  62914527        32         Sec GPT table
  62914559         1         Sec GPT header
# gpt show /dev/zvol/dsk/pail/nbsd
GPT not found, displaying data from MBR.

    start     size  index  contents
        0        1         MBR
        1       62         Unused
       63  8388545      1  MBR part 169 (active)
...

whereas disklabel/fdisk/dkctl/vnconfig apparently are not able:
...
# disklabel  /dev/zvol/dsk/pail/nbsd
disklabel: ioctl DIOCGDINFO: Inappropriate ioctl for device
# vnconfig -c vnd0 /dev/zvol/dsk/pail/nbsd
vnconfig: /dev/rvnd0: VNDIOCSET: Operation not supported
.....

Is there any particular reason for this or is a matter of adding
support for ZFS/ZVols to both? I suppose I could iSCSI-share them and
then connect, but it looks somewhat convoluted...

Chavdar

(apologies for the incorrect 'reply' address earlier).


BTW the iSCSI route to access the zvols from the NetBSD system itself
actually works fine.

Chavdar

On Thu, 15 Aug 2019 at 19:04, Chavdar Ivanov <ci4ic4%gmail.com@localhost> wrote:
>
> Ok, thanks. I didn’t have any idea about that.
>
> On Thu, 15 Aug 2019 at 18:31, Jason Mitchell <jmitchel%bigjar.com@localhost> wrote:
>>
>> Hello,
>>
>> Just as an FYI, I’ve done lots of testing with powerline and you have to divide the number they give you by 7. They send the same data 7 times, so the adapter could send 1 Gbit, but you’ll only ever get 142Mbit.
>>
>> Thanks,
>>
>> Jason M.
>>
>> Sent from my iPhone
>>
>> > On Aug 15, 2019, at 9:14 AM, Chavdar Ivanov <ci4ic4%gmail.com@localhost> wrote:
>> >
>> >> On Thu, 15 Aug 2019 at 13:53, Chavdar Ivanov <ci4ic4%gmail.com@localhost> wrote:
>> >>
>> >> FYI - as I mentioned it elsewhere - I modified my qemu-nvmm VMs from
>> >> GPT slices and raw files to zvols without any problems. My next test
>> >> would be to see if I can ISCSI-share a zvol - which should work.
>> >
>> > It does, as expected, but is a bit slow. I tested using something
>> > called IsMyHDOk, the resulting benchmark was 17.7; using a ZVOL
>> > FreeNAS target over the same slowish connection I get 42.2; for
>> > comparison on the same laptop the second rusty SATA HDD gives me
>> > 175.7, whilst the first M.2 SATA gives me  972.1.
>> >
>> > It is in no way the right way to use iSCSI - the laptop is connected
>> > wirelessly to an access point; the adapter clams current speed 866.7
>> > mb/s (?), which then links to the upstairs room via a powerline
>> > adapter, advertised az 1gb/s, but which has never given me anything
>> > better than about 130mb/s - but then the link to the FreeNAS zvol goes
>> > through the same route and on top of this FreeNAS is a XCP-NG guest,
>> > whereas the NetBSD laptop is using a gigabit Ethernet to a local
>> > switch beyond the powerline adapter...
>> >
>> >
>> >> Later.
>> >>
>> >>> On Thu, 15 Aug 2019 at 13:25, Brad Spencer <brad%anduin.eldar.org@localhost> wrote:
>> >>>
>> >>>
>> >>> I did some more ZFS testing in 9.0_BETA, sorry for the length.  There
>> >>> are a lot of advantages and use cases in getting ZFS more completely
>> >>> integrated into the NetBSD universe.  Raidframe+LVM is great too, but
>> >>> there are things that are harder to do with it.
>> >>>
>> >>> One of the purposes of these tests was to see how NetBSD ZFS interacted
>> >>> with other ZFS implementations in other OSs.  The following was done:
>> >>>
>> >>> "zfs export pool" created from NetBSD and "zfs import pool" into FreeBSD
>> >>> and SmartOS.  Both of these worked completely as expected. with full
>> >>> read and write on the pools.  There are ZFS features not enabled on
>> >>> pools created with NetBSD, but this does not appear to be a problem.  As
>> >>> long as a "zfs upgrade pool" is not performed you can import and export
>> >>> as much as you wish.  I have not tried any Linux implementations.
>> >>>
>> >>> Going the other direction did not work quite as expected.
>> >>>
>> >>> On pools created on FreeBSD, the ZFS features that are turned on will
>> >>> not allow the pool to import as writable on NetBSD.  This is not really
>> >>> an unexpected thing, given that features on turned on in FreeBSD that
>> >>> are not in NetBSD.  The pool will import readonly and appears to work as
>> >>> expected in that case.  It should be possible to create pools in FreeBSD
>> >>> with features disabled that will allow a better import.
>> >>>
>> >>> For SmartOS, pools created there will (might??  I only use the
>> >>> filesystems I presented to the ISO) have a GPT header attached to them.
>> >>> When NetBSD sees that, by default, it will create wedges.  As a general
>> >>> thing, this wouldn't be a problem.  However, "zpool import" does not
>> >>> appear to see dk devices or refuses to deal with them and thus no pools
>> >>> were noticed for import.  You can sort of force the issue by creating a
>> >>> directory with symlinks to the /dev/dkN devices in them and then do a
>> >>> "zpool import -d /dev/directory".  When this is attempted, the DOMU
>> >>> guest I was using panicked:
>> >>>
>> >>> panic: kernel diagnostic assertion "seg <= BLKIF_MAX_SEGMENTS_PER_REQUEST" failed: file "/usr/src/sys/arch/xen/xen/xbd_xenbus.c", line 1032
>> >>>
>> >>> I am running an older 8.99.xx with Xen 4.8.x, but as this was from the
>> >>> guest, I suspect that a bug is still present.  From the BT, it appears
>> >>> that it tried to do the import, but failed somewhere in the lower level
>> >>> disk routines.  I did not try doing this with FreeBSD.
>> >>>
>> >>>
>> >>> Other things of note:
>> >>>
>> >>> Assuming you want to have separate /var and /usr, trying to use ZFS for
>> >>> those is a challenge.  One of the biggest issues is that /zbin/zfs and
>> >>> /zbin/zpool have linked against shared libraries in /usr/lib.  This is
>> >>> probably not for the best.  Further, mostly because of this, you can't
>> >>> reasonably have a /etc/rc.d/zfs start the module and make it usable for
>> >>> critical_filesystems in /etc/rc.conf (you can fake it out by putting the
>> >>> contents of the shared libraries in /usr/lib and then mount over it, but
>> >>> that is a bit ugly).  Having /var and /usr use LVM is possible mostly
>> >>> because /etc/rc.d/lvm exists and is started before critical_filesystems.
>> >>> ZFS should probably work the same way.
>> >>>
>> >>> Another item is that mountpoint=legacy really does not work.  Right now,
>> >>> there is neither support in the general mount system call for ZFS nor
>> >>> does a /sbin/mount_zfs exist.  You can actually get some of this effect
>> >>> by a simple shell script in /sbin called mount_zfs that translates the
>> >>> request into a "/sbin/zfs set mountpoint=XXX" call followed by a "zfs
>> >>> mount XXX".  I wrote a quick and dirty one of these, and with a quick
>> >>> and dirty /etc/rc.d/zfs, it will allow the effect (mostly) having /var
>> >>> and /usr be from a ZFS pool.
>> >>>
>> >>> An unexpected problem was with /var in this configuration, however.  For
>> >>> reasons that are not entirely obvious postfix will not start if /var is
>> >>> from a ZFS pool.  It errors in an unexpected way when trying to start
>> >>> postfix master.  I did not have time to do a full ktrace to see what it
>> >>> was trying to use, but something is not quite supported, or not
>> >>> supported in the same way as a FFS.
>> >>>
>> >>> An alternative might be to use a zvol with a FFS in it for /var.  This
>> >>> mostly does work as expected from the command line interactively.
>> >>> However, it won't work from boot because of a couple of problems.
>> >>>
>> >>> 1) The names of the devices confuse fsck and it fails on boot.  The zvol
>> >>> will be called something like /dev/zvol/dsk/pool/volname in fstab and
>> >>> fsck will want to try and clean /dev/zvol/dsk/pool/rvolname which is
>> >>> wrong for ZFS zvols.  You need to use /dev/zvol/rdsk/pool/volname.  You
>> >>> can put a symlink in and it will mostly work, but that is highly manual.
>> >>> LVM has a symlink tree it creates that points the devices at the right
>> >>> places in /dev/mapper which allows fsck and mount to work pretty much as
>> >>> needed.  We could something like that for ZFS or teach fsck about zvol
>> >>> names.
>> >>>
>> >>> 2) Suppose you wanted to avoid fsck by trying to use wapbl logging.
>> >>> There are missing IOCTLs that will prevent the log directive from
>> >>> working, so "mount -o log /dev/zvol/fsk/pool/volname" will not work.
>> >>>
>> >>> The end result is that while you can define a zvol and put a FFS on it,
>> >>> you can't clean it on boot and you can't use FFS wabpl.
>> >>>
>> >>>
>> >>> Disklabel won't work with zvols, but this is not really fatal.  I don't
>> >>> think they work with LVM volumns either.  They make little sense in
>> >>> either case.  You can put a GPT label on a zvol, but dkctl can't create
>> >>> wedges due to missing IOCTLs.  I never tried doing this with a LVM
>> >>> volumn, so I don't really know what happens in that case.
>> >>>
>> >>> You can not use a zvol for swap.  Again, probably not fatal, as I don't
>> >>> think you can use a LVM volumn for swap either, but this may stand in
>> >>> the way of making the root filesystem a ZFS filesystem.
>> >>>
>> >>>
>> >>>
>> >>> All in all, however, this all does work pretty nicely for a lot of
>> >>> things one would want to do.  I will probably end up moving my Xen disks
>> >>> from raidframe+lvm to zvols.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Brad Spencer - brad%anduin.eldar.org@localhost - KC8VKS - http://anduin.eldar.org
>> >>
>> >>
>> >>
>> >> --
>> >> ----
>> >
>> >
>> >
>> > --
>> > ----
>>
> --
> ----



--
----


-- 
----