Re: Network traffic performance drops when two domUs share the pipe.

[Brad Spencer <brad%anduin.eldar.org@localhost>]

"Mathew, Cherry G." <c%bow.st@localhost> writes:

> On May 22, 2023 1:26:38 PM UTC, Manuel Bouyer <bouyer%antioche.eu.org@localhost> wrote:
>>On Mon, May 22, 2023 at 04:25:54AM +0000, Mathew, Cherry G.* wrote:
>>> Hi There,
>>> 
>>> I have an outbound pipe that can hit about 1MBps downstream on dom0 or a
>>> single pv domU.
>>> 
>>> When I run a hvm domU as well, the total bandwidth performance of both
>>> domUs combined drops to about 20-30% (I don't have exact numbers as I
>>> haven't figured out a way to measure this), and sometimes, I notice that
>>> the connections from one or the other domU gets dropped.
>>> 
>>> This is on NetBSD 10.0_BETA dom0 with a NetBSD 9.3 pv domU and a FreeBSD
>>> 13.1-RELEASE hvm domU.
>>> 
>>> I'm wondering if anyone else is seeing this behaviour.
>>
>>1MBps doesn't look that big. I have several Xen servers with gigabit
>>connectivity, and multiple domUs and didn't notice such issues.
>>
>
> Perhaps the slower speed makes it noticeable. In my case , traffic on one domU seems to completely degrade traffic on the other. 
>
> I need to figure out a way to test this more specifically.
>
>
>>Do you have some kind of firewall running on the dom0 ?
>>
>
> Not on this host, although the outbound gateway has a standard npf(7) NAT.


I would check sndq and rcvq drops on both the DOM0 and DOMU.

I had to increase IFQ_MAXLEN a lot on a couple of my DOM0 systems to
keep drops from happening when I went to NetBSD >= 9.0.  What I noticed
was problems with drops when a DOMU was a router for a lot of networks.
If the DOM0 started to drop (say because of pressure from the DOMU, the
DOMU would start to drop and if enough of this sort of thing happened,
it would cause trouble with higher protocols.  In a couple of cases I
also increased IFQ_MAXLEN on DOMUs.

There is a sysctl that allows this to be increased for sndq, but not one
currently for rcvq and that one was where I had a lot of the problems.

All DOM0s have older wm(4) physical cards running at 1Gbit with 2 to 5
wm(4) ports across multiple cards.  Multiple VLANs in play.




-- 
Brad Spencer - brad%anduin.eldar.org@localhost - KC8VKS - http://anduin.eldar.org