xennet slow under -current

[marcotte%panix.com@localhost (Brian Marcotte)]

I've been seeing poor xennet performance under -current for some time
now. I've done some tests which show some interesting results.

  * Input wasn't great in 9.1, but in -current, it's even slower.
  * PVH(VM) slows output by a factor of 5.
  * Running against "lo0" also shows large performance loss on -current
  * No change for e1000 (PCI passthrough).
  * Virtio has improved in -current.

    O/S     type    int      out       in (MBits/s)
    ------- ----    -----   ----    -----
    9.1     PV      xennet  1900      300
    current PV      xennet  1960       45
    current PVH     xennet   382       42
    current PVHVM   xennet   344       44

    9.1     PVHVM   wm       527      567
    current PVHVM   wm       515      530

    9.1     HVM     vioif    271      376
    current HVM     vioif    450      540

    9.1     PV      lo      9580    10600
    9.1     HVM     lo      4800     5280
    current PV      lo      6500     6210
    current PVH     lo      3950     3700
    current HVM     lo      3980     3940

       wm = PCI passthrough of "e1000". I had to set WM_DISABLE_MSIX=1
            for it to work reliably.

Host system is Intel E5-2667 v2, Xen 4.14 and Linux 4.19. Tests were
done with iperf3 and TCP. iperf3 server is on FreeBSD running directly
on hardware.

Let me know if I can help debug this further.

Thanks.

--
- Brian