Re: nvmm users - experience

[Reinoud Zandijk <reinoud%NetBSD.org@localhost>]

Hi,

On Sun, May 21, 2023 at 02:01:22PM +0000, Mathew, Cherry G.* wrote:
> I'm wondering if there are any nvmm(4) users out there - I'd like to
> understand what your user experience is - expecially for multiple VMs
> running simultaneously.
> 
> Specifically, I'd like to understand if nvmm based qemu VMs have
> interactive "jitter" or any such scheduling related effects.
> 
> I tried 10.0_BETA with nvmm and it was unusable for 3guests that I
> migrated from XEN, so I had to fall back.
> 
> Just looking for experiences from any users of nvmm(4) on NetBSD (any
> version, including -current is fine).

How many CPUs does each instance have and how many CPUs does your machine
have? It could be, but thats just a hunch reading the code since I'm not that
deep in NVMM (!), that possibly more than one Qemu has vCPUs activated and all
host CPUs are busy? This will mean that the other Qemu processes are not run
(for I/O etc) until a tick interrupt comes by or an userland event like IO
read/pagefault comes by and one vCPU is aborted. This will be handled in the
kernel and control will be given back to its nvmm userland part and the qemu
handler might interrupt the other vCPUs to trigger their virtual interrupt
handling. Note that while running a vCPU the kernel preemption is switched
off.

This might give the irregular IO performance you see as the qemu IO threads
are not sheduled often enough.

A solution might be that the nvmm code in *_vcpu_run() sets a timer interrupt
with a dummy handler on its CPU and on its forced vmreturn checks the number
of cpus currently not running NVMM cpus and if that number is lower than the
number of virtual machines that are declared and running it yields by
returning to userland to have at least one CPU for each of the Qemus.
Otherwise it can continue as before without making the expensive switch back
to userland.

Ideally we ought to ask Maxime Villard who wrote the code and check/coordinate
with DragonFlyBSD but he also isn't with DragonFly anymore IIRC.

So, I might be barking up the wrong tree but this might be it.

With regards,
Reinoud