CVS commit: src

["Maxime Villard" <maxv%netbsd.org@localhost>]

Module Name:    src
Committed By:   maxv
Date:           Sun Apr 28 14:22:14 UTC 2019

Modified Files:
        src/lib/libnvmm: Makefile libnvmm.c libnvmm_x86.c nvmm.h
        src/sys/dev/nvmm: nvmm.c nvmm.h nvmm_internal.h nvmm_ioctl.h
        src/sys/dev/nvmm/x86: nvmm_x86.h nvmm_x86_svm.c nvmm_x86_vmx.c

Log Message:
Modify the communication layer between the kernel NVMM driver and libnvmm:
introduce a bidirectionnal "comm page", a page of memory shared between
the kernel and userland, and used to transfer data in and out in a more
performant manner than ioctls.

The comm page contains the VCPU state, plus three flags:

 - "wanted": the states the kernel must get/set when requested via ioctls
 - "cached": the states that are in the comm page
 - "commit": the states the kernel must set in vcpu_run

The idea is to avoid performing expensive syscalls, by using the VCPU
state cached, either explicitly or speculatively, in the comm page. For
example, if the state is cached we do a direct 1->5 with no syscall:

          +---------------------------------------------+
          |                    Qemu                     |
          +---------------------------------------------+
               |                                   ^
               | (0) nvmm_vcpu_getstate            | (6) Done
               |                                   |
               V                                   |
             +---------------------------------------+
             |                libnvmm                |
             +---------------------------------------+
                  |   ^          |               ^
        (1) State |   | (2) No   | (3) Ioctl:    | (5) Ok, state
        cached?   |   |          | "please cache | fetched
                  |   |          |  the state"   |
                  V   |          |               |
              +-----------+      |               |
              | Comm Page |------+---------------+
              +-----------+      |
                       ^         |
          (4) "Alright |         V
               babe"   |     +--------+
                       +-----| Kernel |
                             +--------+

The main changes in behavior are:

 - nvmm_vcpu_getstate(): won't emit a syscall if the state is already
   cached in the comm page, will just fetch from the comm page directly
 - nvmm_vcpu_setstate(): won't emit a syscall at all, will just cache
   the wanted state in the comm page
 - nvmm_vcpu_run(): will commit the to-be-set state in the comm page,
   as previously requested by nvmm_vcpu_setstate()

In addition to this, the kernel NVMM driver is changed to speculatively
cache certain states known to be of interest, so that the future
nvmm_vcpu_getstate() calls libnvmm or the emulator will perform will use
the comm page rather than expensive syscalls. For example, if an I/O
VMEXIT occurs, the I/O Assist in libnvmm will want GPRS+SEGS+CRS+MSRS,
and now the kernel caches all of that in the comm page before returning
to userland.

Overall, in a normal run of Windows 10, this saves several millions of
syscalls. Eg on a 4CPU Intel with 4VCPUs, booting the Win10 install ISO
goes from taking 1min35 to taking 1min16.

The libnvmm API is not changed, but the ABI is. If we changed the API it
would be possible to save expensive memcpys on libnvmm's side. This will
be avoided in a future version. The comm page can also be extended to
implement future services.


To generate a diff of this commit:
cvs rdiff -u -r1.5 -r1.6 src/lib/libnvmm/Makefile
cvs rdiff -u -r1.9 -r1.10 src/lib/libnvmm/libnvmm.c src/lib/libnvmm/nvmm.h
cvs rdiff -u -r1.28 -r1.29 src/lib/libnvmm/libnvmm_x86.c
cvs rdiff -u -r1.18 -r1.19 src/sys/dev/nvmm/nvmm.c
cvs rdiff -u -r1.7 -r1.8 src/sys/dev/nvmm/nvmm.h
cvs rdiff -u -r1.9 -r1.10 src/sys/dev/nvmm/nvmm_internal.h
cvs rdiff -u -r1.5 -r1.6 src/sys/dev/nvmm/nvmm_ioctl.h
cvs rdiff -u -r1.12 -r1.13 src/sys/dev/nvmm/x86/nvmm_x86.h
cvs rdiff -u -r1.42 -r1.43 src/sys/dev/nvmm/x86/nvmm_x86_svm.c
cvs rdiff -u -r1.30 -r1.31 src/sys/dev/nvmm/x86/nvmm_x86_vmx.c

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.