Re: PCI passthrough not working with bhyve and NetBSD

I have current installed on it now.

It has both the Intel and Broadcom NICs with passthrough as well as the virtual NIC from bhyve.

The virtual NIC works nice. I used it to fetch the installation sets.

From dmesg:

[ 1.0448813] virtio1 at pci0 dev 5 function 0

[ 1.0448813] virtio1: Virtio Network Device (rev. 0x00)

[ 1.0448813] vioif0 at virtio1: Features: 0x11010020<INDIRECT_DESC,NOTIFY_ON_EMPTY,STATUS,MAC>

[ 1.0448813] vioif0: Ethernet address 58:9c:fc:0b:96:50

[ 1.0448813] virtio1: config interrupting at msix0 vec 0

[ 1.0448813] virtio1: queues interrupting at msix0 vec 1

[ 1.0448813] wm0 at pci0 dev 6 function 0: 82576 quad-1000BaseT Ethernet (rev. 0x01)

[ 1.0448813] wm0: interrupting at msi1 vec 0

[ 1.0448813] wm0: Ethernet address 00:1b:21:84:e9:ed

[ 1.0448813] igphy0 at wm0 phy 1: i82566 10/100/1000 media interface, rev. 1

[ 1.0448813] igphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto

[ 1.0448813] bge0 at pci0 dev 7 function 0: Broadcom BCM5720 Gigabit Ethernet

[ 1.0448813] bge0: APE firmware NCSI 1.4.12.0

[ 1.0448813] bge0: interrupting at msi2 vec 0

[ 1.0448813] bge0: HW config 002b1094, 00006014, 0002aa38, 00000000 0000000c

[ 1.0448813] bge0: ASIC BCM5720 A0 (0x5720000), Ethernet address 3c:a8:2a:e4:bc:eb

[ 1.0448813] bge0: no PHY found!

[ 1.0448813] pcib0 at pci0 dev 31 function 0: vendor 8086 product 7000 (rev. 0x00)

[ 1.0448813] isa0 at pcib0

[ 1.0448813] com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo

[ 1.0448813] com0: console

[ 1.0448813] com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo

[ 1.5402764] WARNING: 1 error while detecting hardware; check system log.

The intel NIC is connected at the moment.

It should receive an address with DHCP.

localhost# ifconfig

vioif0: flags=0x8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500

ec_capabilities=1<VLAN_MTU>

ec_enabled=0

address: 58:9c:fc:0b:96:50

inet6 fe80::5a9c:fcff:fe0b:9650%vioif0/64 flags 0x0 scopeid 0x1

wm0: flags=0x8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

capabilities=7ff80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx>

capabilities=7ff80<TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Rx>

capabilities=7ff80<TCP6CSUM_Tx,UDP6CSUM_Rx,UDP6CSUM_Tx,TSO6>

enabled=0

ec_capabilities=7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>

ec_enabled=0

address: 00:1b:21:84:e9:ed

media: Ethernet autoselect (1000baseT full-duplex,flowcontrol,rxpause,txpause)

status: active

inet 169.254.58.45/16 broadcast 169.254.255.255 flags 0x0

inet6 fe80::220e:9ccf:9196:97d1%wm0/64 flags 0x0 scopeid 0x2

bge0: flags=0x8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500

capabilities=3f80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx>

capabilities=3f80<UDP4CSUM_Rx,UDP4CSUM_Tx>

enabled=0

ec_capabilities=7<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU>

ec_enabled=0

address: 3c:a8:2a:e4:bc:eb

media: Ethernet manual (none)

lo0: flags=0x8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33624

inet 127.0.0.1/8 flags 0x0

inet6 ::1/128 flags 0x20<NODAD>

inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x4

localhost#

rc.conf:

dhcpcd=YES

dhcpcd_flags="-qM wm0"

Any help would be much appreciated.

If I'm able to get one of the physical NICs to work somehow, I could give anyone here willing to help debug SSH access to the server.

On Sun, Jul 15, 2018 at 1:49 PM <gary%duzan.org@localhost> wrote:

"Farid Joubbi" <djfarid%gmail.com@localhost> wrote:
=> I tried the install CD for the latest HEAD-201807121210Z.
=> It can see both the NICs, but they don't work.
=> Trying to configure them with DHCP, it looks as if the NIC is sending a
=> DISCOVERY but timing out never receiving anything back (I doubt that it is
=> actually sending anything on the wire).
=> Then it constantly throws this on the console: "wm0: device timeout (lost
=> interrupt)".

That sounds like the kernel missing interrupts from the NIC, in which
case it may actually be sending out the requests but failing to see the
replies. Perhaps posting the dmesg, or at least the lines about the
NICs, could help.

Gary Duzan

=> On Sat, Jul 14, 2018 at 8:34 PM Farid Joubbi <djfarid%gmail.com@localhost> wrote:
=>
=>> I have a working configuration for NetBSD on bhyve. It's not a problem
=>> with vm-bhyve https://github.com/churchers/vm-bhyve
=>>
=>> NetBSD 7.1.2 does not initiate the bge0 nor wm0 at all under bhyve, as I
=>> wrote earlier.
=>> It seems to run perfectly fine other than that.
=>> The "normal" vioif0 works fine.
=>>
=>> I tried NetBSD 8.0 RC2.
=>> It finds both NICs! All of a sudden I feel hope ;-)
=>> Unfortunately I can only get link-local addresses on both NICs.
=>> And I also get this printed on the console every now and then: "bge0:
=>> watchdog timeout -- resetting".
=>>
=>> I will try and get it to NetBSD-current and see.
=>>
=>>
=>> On Tue, Jul 10, 2018 at 5:30 AM Travis Paul <Tr%vispaul.me@localhost> wrote:
=>>
=>>>
=>>>
=>>> > On 9 Jul 2018, at 5:58 AM, Farid Joubbi <djfarid%gmail.com@localhost> wrote:
=>>> >
=>>> > Thanks for the reply.
=>>> > After reading it, I realize that the learning curve for me to
=>>> understand what is going on is a bit too steep.
=>>> > I know only some basic C programming from university courses several
=>>> years ago.
=>>> > This kind of learning was not what I had in mind when I figured that
=>>> I
=>>> want to run a new NetBSD installation this summer... ;-(
=>>>
=>>> FWIW, NetBSD runs fine on Bhyve without PCI passthru. I'm running a few
=>>> NetBSD 7 and 8 VMs on FreeBSD 11.
=>>>
=>>> I can share some commands to get a NetBSD system up if you're
=>>> interested
=>>> but I'm not using PCI passthru so I can't assist there.
=>>>
=>>> Best,
=>>> Travis
=>>>
=>>
=>