Re: Dom0 bad network performance (was: NetBSD/Xen samba performance low (compared to NetBSD/amd64))

[Thor Lancelot Simon <tls%panix.com@localhost>]

It is much more likely the hardware -- this is a Realtek adapter, after all
(pretty surprising to see Intel put one of those in a NUC, but, maybe they
have finally decided to entirely exit the 1Gbit NIC business?).

I bet interrupts aren't arriving when you expect -- flat out being missed --
and subsequent trips into the interrupt handler pick up the packets anyway.
A useful experiment might be to _disable_ interrupt coalescing and see whether
link utilization goes up along with the inevitable increase in CPU
utilization.

Also, at least some Realtek adapters have had really nasty interactions
between protocol offload and interrupt delivery.  You should test the various
combinations of offload features the adapter and driver support - I assume
your baseline testing has them all turned on?  It would be particularly
interesting to see the effect of turning large send (segmentation offload)
off and on since, if it goes wrong, this can have the side effect of holding
or dropping acks on some adapters, and thus pacing not just send, which might
be expected, but also receive, which often might not,

I have encountered at least one Realtek adapter where if any offload feature
was turned on, I actually had to poll to not miss RX interrupts.  But that
was long ago, and I would hope they are better now,

On Sat, Feb 13, 2021 at 10:52:05AM +0100, Manuel Bouyer wrote:
> On Sat, Feb 13, 2021 at 05:03:59AM +0100, Matthias Petermann wrote:
> > Hello all,
> > 
> > last summer I already had a request here, which I had classified as a Samba
> > problem at that time due to insufficient preliminary investigations. You
> > guys helped me a lot to investigate the problem further, but unfortunately I
> > didn't have the chance to follow up on it.
> > 
> > Now I have made a new attempt on the same hardware (Intel NUC5) and have
> > gone a little deeper into the matter. My new findings are:
> > 
> >  - the raw read/write performance in the filesystem is hardly different, no
> > matter if I use NetBSD as Dom0 or on the same system a "pure" NetBSD kernel
> > without Xen. In both cases it is about 60 MByte/s which appears ok for this
> > type of hardware.
> > 
> >  - when transferring files of this file system over the network the
> > performance is lost - with NetBSD as Dom0 the transfer rate is max. 12
> > MByte/s, while it is about 60 MByte/s when using a "pure" NetBSD kernel
> > without Xen. It does not matter whether the transfer is done with Samba or
> > another protocol (sftp, NFS).
> 
> I noticed this too, but never took the time to investigate.
> At first glance it could be an issue with latency, which is highter
> with a dom0 setup than with bare metal (the kernel/userland boundary is
> much more costly for dom0).
> 
> -- 
> Manuel Bouyer <bouyer%antioche.eu.org@localhost>
>      NetBSD: 26 ans d'experience feront toujours la difference
> --

-- 
 Thor Lancelot Simon	                                     tls%panix.com@localhost
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."	--Andrei Sakharov