port-xen/57121: Pure NetBSD PVH guests do not appear to be able to use more than 4GB of memory

[brad%anduin.eldar.org@localhost]

>Number:         57121
>Category:       port-xen
>Synopsis:       Pure NetBSD PVH guests do not appear to be able to use more than 4GB of memory
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    port-xen-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Dec 19 15:20:00 +0000 2022
>Originator:     brad%anduin.eldar.org@localhost
>Release:        NetBSD 10.0_BETA
>Organization:
	eldar.org
>Environment:
System: NetBSD nbsd10test 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Sat Dec 17 10:31:04 EST 2022  brad%samwise.nat.eldar.org@localhost:/lhome/NetBSD_10_branch_20221216/amd64/OBJ/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:

With a pure PVH 64bit NetBSD guest using the GENERIC kernel, the guest
does not appear to be able to use more than 4GB of memory.  That is,
no matter how much memory you give the guest the guest caps it at 4GB.
This does not happen with ArchLinux guests on the same DOM0.  The same
guest configured as a PV system with pvshim sees all of the memory.

>How-To-Repeat:

Simple config for the PVH guest with redacted MAC address:

kernel = "/lhome/xen/kernels/NetBSD_10.x/NetBSD_10_branch_20221216/amd64/netbsd-GENERIC"
memory = 8192
cpu_weight = 32
type="pvh"
name = "nbsd10test"
vcpus = 1
vif = [ 'mac=WH:AT:TE:VE:R0:00, bridge=bridge4' ]
disk = [ 'phy:/dev/mapper/rustvg0-nbsd10testlv0,0x01,w','phy:/dev/mapper/rustvg0-nbsd10testlv1,0x02,w','phy:/dev/mapper/rustvg0-nbsd10testlv2,0x03,w','phy:/dev/mapper/rustvg0-nbsd10testlv3,0x04,w' ]
root = "xbd0"

From the DOM0:

DOM0# xl list nbsd10test -v
Name                                        ID   Mem VCPUs      State   Time(s)   UUID                            Reason-Code                                                     Security Label
nbsd10test                                  33  8191     1     -b----       1.4 e7c5bab3-2bd9-4994-9ef9-b80af5e0b927        -                -


From the guest:

nbsd10test# dmesg|grep memory
[     1.000000] total memory = 4096 MB
[     1.000000] avail memory = 3936 M
nbsd10test# cat /proc/meminfo
        total:    used:    free:  shared: buffers: cached:
Mem:  4137267200 150634496 3986632704        0 11395072 56442880
Swap: 10737414144        0 10737414144
MemTotal:   4040300 kB
MemFree:    3893196 kB
MemShared:        0 kB
Buffers:    3893196 kB
Cached:       55120 kB
SwapTotal: 10485756 kB
SwapFree:  10485756 kB


The DOM0 is a NetBSD/amd64 system running 9.3_STABLE with
xenkernel415-4.15.3 from pkgsrc 2022Q3.  As mentioned ArchLinux PVH
guests see all of the memory given to them.

>Fix:

You can work around this by running the guest as a PV guest with or
without pvshim.  However, the desire, or need, to run the guest in
pure PVH mode will likely become more desirable or required as time
goes on.