Re: PR/55822 CVS commit: src/sys/dev/pci

[Bert Kiers <kiers%original.xs4all.nl@localhost>]

On Wed, Dec 02, 2020 at 10:45:40AM +0900, s ymgch wrote:
> Hi,

Hi,

> The reason why iavf(4) did not work on bhyve may be the combination between
> bhyve and NetBSD, because it didn't appear on ESXi and Linux as far as I know.
> 
> Can you let me know about the environment of the VM?
> 
> 1. Which version of bhyve did you use?

The one that comes with FreeBSD 12.2-RELEASE r366954.
bhyve nor bhyvectl have an option that shows the version, afaik.

> 2. Which driver version of ixl did the bhyve use?
>    - This is not the FW version shown in a probe message.

I see

ixl0: <Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.0-k> mem 0xc4000000-0xc4ffffff,0xc5008000-0xc500ffff irq 36 at device 0.0 numa-domain 0 on pci8

so 2.3.0-k

On

https://downloadcenter.intel.com/download/25160/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connection-under-FreeBSD-

I see version 1.12.3.  I'll go try that.  But I do not understand how
2.3.0-k relates to 1.12.3 yet.

> 3. Can other network devices using PCI passthrough, for example ixl, ixg and ixv
>    work well on the same VM (not NetBSD-9 host)

I just have ixl NICs on this machine.  So I don't know.

(In an other server I have a X540 NIC, that is ix on the FreeBSD side
and ixv in the NetBSD guest.  That works stable and fast, but just for
*one* guest, I documented that story on
https://netbsd.itsx.net/freebsd+bhyve+sriov.html, which is work in 
progress).

On this machine with ixl, multiple FreeBSD and Linux guests work.


> And can you tell me the message when iavf(4) is loaded with -i
> debug_level=3 option?
> You may already know, but just in case, the usage of the debug option
> is following:
> 
> 1. build and install kernel not containing iavf(4)
> 2. build and install modules
> 3. run the kernel and do "modload -i debug_level=3 if_iavf" command

Boot messages:

[     1.033888] Intel XL710 Ethernet Virtual Function (ethernet network, revision 0x01) at pci0 dev 2 function 0 not configured
[     1.033888] Intel X722 10GbE VF (ethernet network, revision 0x09) at pci0 dev 3 function 0 not configured

and then

# modload -i debug_level=3 if_iavf
[  61.5983006] iavf: debug level=3
[  61.6083181] iavf0 at pci0 dev 2 function 0iavf0: post
[  61.6283146] iavf0: flags 0x3400<SI,BUF,DB> opcode 0801
[  61.6283146] iavf0: datalen 8 retval 0
[  61.6283146] iavf0: vc-opcode 1 (GET_VERSION)
[  61.6381634] iavf0: vc-retval 0
[  61.6381634] iavf0: cookie 0000000000000001
[  61.6381634] iavf0: 00000000 00000000 00000001 07fd0000
[  61.9183239] iavf0: atq timedout
:unable to get VF interface version
[  61.9379397] iavf1 at pci0 dev 3 function 0iavf1: post
[  61.9483230] iavf1: flags 0x3400<SI,BUF,DB> opcode 0801
[  61.9483230] iavf1: datalen 8 retval 0
[  61.9483230] iavf1: vc-opcode 1 (GET_VERSION)
[  61.9483230] iavf1: vc-retval 0
[  61.9582973] iavf1: cookie 0000000000000001
[  61.9582973] iavf1: 00000000 00000000 00000001 080d5000
[  62.2182973] iavf1: atq timedout
:unable to get VF interface version


I'll put complete dmesg and things on
https://netbsd.itsx.net/PR55822/ soon.

Grtnx,
--
Bert