Re: PCI passthrough not working with bhyve and NetBSD

[Jaromír Doleček <jaromir.dolecek%gmail.com@localhost>]

MSI and MSI-X themselves are supported on amd64, though not every
driver use it. Both wm(4) and bge(4) however do support it and use it,
so should work.

Maybe there is either some misunderstanding about the MSI support
(e.g. our PCI subsystem doesn't detect it), or the PCI probe code
tries to allocate (and fails) the INTx interrupt regardless of MSI.

Jaromir

2018-08-10 0:34 GMT+02:00 Farid Joubbi <djfarid%gmail.com@localhost>:
> I think I know the cause of the problem.
> I posted this same question on the FreeBSD virtualization list, and received
> this reply which sounds plausible:
>
> You have to use MSI interrupts for pass-through devices for bhyve.  Those
> errors look as if the guests are trying to use legacy INTx interrupts.
>
> I found this paper explaining the problem with the lack of MSI-X support:
>
> https://www.netbsd.org/gallery/presentations/ozaki-r/2015_AsiaBSDCon/ABC2015-P4C-paper.pdf
>
> The paper is three years old. What is the state of MSI in NetBSD now?
> Is my problem due to lack of MSI support, or is the support there but not
> used?
>
>
> On Sun, Jul 15, 2018, 21:00 Farid Joubbi <djfarid%gmail.com@localhost> wrote:
>>
>> I have current installed on it now.
>> It has both the Intel and Broadcom NICs with passthrough as well as the
>> virtual NIC from bhyve.
>> The virtual NIC works nice. I used it to fetch the installation sets.
>>
>> From dmesg:
>> [   1.0448813] virtio1 at pci0 dev 5 function 0
>> [   1.0448813] virtio1: Virtio Network Device (rev. 0x00)
>> [   1.0448813] vioif0 at virtio1: Features:
>> 0x11010020<INDIRECT_DESC,NOTIFY_ON_EMPTY,STATUS,MAC>
>> [   1.0448813] vioif0: Ethernet address 58:9c:fc:0b:96:50
>> [   1.0448813] virtio1: config interrupting at msix0 vec 0
>> [   1.0448813] virtio1: queues interrupting at msix0 vec 1
>> [   1.0448813] wm0 at pci0 dev 6 function 0: 82576 quad-1000BaseT Ethernet
>> (rev. 0x01)
>> [   1.0448813] wm0: interrupting at msi1 vec 0
>> [   1.0448813] wm0: Ethernet address 00:1b:21:84:e9:ed
>> [   1.0448813] igphy0 at wm0 phy 1: i82566 10/100/1000 media interface,
>> rev. 1
>> [   1.0448813] igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
>> 1000baseT, 1000baseT-FDX, auto
>> [   1.0448813] bge0 at pci0 dev 7 function 0: Broadcom BCM5720 Gigabit
>> Ethernet
>> [   1.0448813] bge0: APE firmware NCSI 1.4.12.0
>> [   1.0448813] bge0: interrupting at msi2 vec 0
>> [   1.0448813] bge0: HW config 002b1094, 00006014, 0002aa38, 00000000
>> 0000000c
>> [   1.0448813] bge0: ASIC BCM5720 A0 (0x5720000), Ethernet address
>> 3c:a8:2a:e4:bc:eb
>> [   1.0448813] bge0: no PHY found!
>> [   1.0448813] pcib0 at pci0 dev 31 function 0: vendor 8086 product 7000
>> (rev. 0x00)
>> [   1.0448813] isa0 at pcib0
>> [   1.0448813] com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
>> [   1.0448813] com0: console
>> [   1.0448813] com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
>> [   1.5402764] WARNING: 1 error while detecting hardware; check system
>> log.
>>
>>
>> The intel NIC is connected at the moment.
>> It should receive an address with DHCP.
>>
>> localhost# ifconfig
>> vioif0: flags=0x8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu
>> 1500
>>         ec_capabilities=1<VLAN_MTU>
>>         ec_enabled=0
>>         address: 58:9c:fc:0b:96:50
>>         inet6 fe80::5a9c:fcff:fe0b:9650%vioif0/64 flags 0x0 scopeid 0x1
>> wm0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>         capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>
>>
>> capabilities=7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>
>>         capabilities=7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>
>>         enabled=0
>>         ec_capabilities=7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
>>         ec_enabled=0
>>         address: 00:1b:21:84:e9:ed
>>         media: Ethernet autoselect (1000baseT
>> full-duplex,flowcontrol,rxpause,txpause)
>>         status: active
>>         inet 169.254.58.45/16 broadcast 169.254.255.255 flags 0x0
>>         inet6 fe80::220e:9ccf:9196:97d1%wm0/64 flags 0x0 scopeid 0x2
>> bge0: flags=0x8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
>>
>> capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>
>>         capabilities=3f80<UDP4CSUM_Rx,UDP4CSUM_Tx>
>>         enabled=0
>>         ec_capabilities=7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>
>>         ec_enabled=0
>>         address: 3c:a8:2a:e4:bc:eb
>>         media: Ethernet manual (none)
>> lo0: flags=0x8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33624
>>         inet 127.0.0.1/8 flags 0x0
>>         inet6 ::1/128 flags 0x20<NODAD>
>>         inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x4
>> localhost#
>>
>> rc.conf:
>> dhcpcd=YES
>> dhcpcd_flags="-qM wm0"
>>
>> Any help would be much appreciated.
>> If I'm able to get one of the physical NICs to work somehow, I could give
>> anyone here willing to help debug SSH access to the server.
>>
>>
>> On Sun, Jul 15, 2018 at 1:49 PM <gary%duzan.org@localhost> wrote:
>>>
>>> "Farid Joubbi" <djfarid%gmail.com@localhost> wrote:
>>> => I tried the install CD for the latest HEAD-201807121210Z.
>>> => It can see both the NICs, but they don't work.
>>> => Trying to configure them with DHCP, it looks as if the NIC is sending
>>> a
>>> => DISCOVERY but timing out never receiving anything back (I doubt that
>>> it is
>>> => actually sending anything on the wire).
>>> => Then it constantly throws this on the console: "wm0: device timeout
>>> (lost
>>> => interrupt)".
>>>
>>>    That sounds like the kernel missing interrupts from the NIC, in which
>>> case it may actually be sending out the requests but failing to see the
>>> replies. Perhaps posting the dmesg, or at least the lines about the
>>> NICs, could help.
>>>
>>>                                  Gary Duzan
>>>
>>>
>>> => On Sat, Jul 14, 2018 at 8:34 PM Farid Joubbi <djfarid%gmail.com@localhost>
>>> wrote:
>>> =>
>>> =>> I have a working configuration for NetBSD on bhyve. It's not a
>>> problem
>>> =>> with vm-bhyve https://github.com/churchers/vm-bhyve
>>> =>>
>>> =>> NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve,
>>> as I
>>> =>> wrote earlier.
>>> =>> It seems to run perfectly fine other than that.
>>> =>> The "normal" vioif0 works fine.
>>> =>>
>>> =>> I tried NetBSD 8.0 RC2.
>>> =>> It finds both NICs! All of a sudden I feel hope ;-)
>>> =>> Unfortunately I can only get link-local addresses on both NICs.
>>> =>> And I also get this printed on the console every now and then: "bge0:
>>> =>> watchdog timeout -- resetting".
>>> =>>
>>> =>> I will try and get it to NetBSD-current and see.
>>> =>>
>>> =>>
>>> =>> On Tue, Jul 10, 2018 at 5:30 AM Travis Paul <Tr%vispaul.me@localhost> wrote:
>>> =>>
>>> =>>>
>>> =>>>
>>> =>>> > On 9 Jul 2018, at 5:58 AM, Farid Joubbi <djfarid%gmail.com@localhost> wrote:
>>> =>>> >
>>> =>>> > Thanks for the reply.
>>> =>>> > After reading it, I realize that the learning curve for me to
>>> =>>> understand what is going on is a bit too steep.
>>> =>>> > I know only some basic C programming from university courses
>>> several
>>> =>>> years ago.
>>> =>>> > This kind of learning was not what I had in mind when I figured
>>> that
>>> =>>> I
>>> =>>> want to run a new NetBSD installation this summer... ;-(
>>> =>>>
>>> =>>> FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a
>>> few
>>> =>>> NetBSD 7 and 8 VMs on FreeBSD 11.
>>> =>>>
>>> =>>> I can share some commands to get a NetBSD system up if you're
>>> =>>> interested
>>> =>>> but I'm not using PCI passthru so I can't assist there.
>>> =>>>
>>> =>>> Best,
>>> =>>> Travis
>>> =>>>
>>> =>>
>>> =>
>>>
>>>
>