kern/57776: Under KVM: [ 1.3711794] uvm_fault(0xffffffff8190fb40, 0xffffaf80b451e000, 2) -> e

[andrew.cagney%gmail.com@localhost]

>Number:         57776
>Category:       kern
>Synopsis:       Under KVM: [   1.3711794] uvm_fault(0xffffffff8190fb40, 0xffffaf80b451e000, 2) -> e
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Dec 15 15:55:00 +0000 2023
>Originator:     Andrew Cagney
>Release:        NetBSD-10.0_RC1-amd64.iso
>Organization:
>Environment:
Linux tundra.libreswan.org 6.5.12-300.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 22 18:12:37 EST 2023 x86_64 GNU/Linux
>Description:
Per above, the KVM host is Fedora, and the h/w is:

  model name      : Intel(R) Xeon(R) w3-2423

The interesting bits are at the end.  But for record:

- there's a stock boot.iso (so console works)
- a modified full iso (some scripts to do the install added and mounted as a second ISO)
- things barf trying to boot the stock boot.iso

good news is it is reproducible.

Given this is working on a number of other machines I tend to suspect the combination of host hardware, kvm, and NetBSD.

make: Entering directory '/home/build/repo-under-test/testing/libvirt'
: kvm-base-netbsd
rm -f /home/build/pool/w1.netbsd-base
rm -f /home/build/pool/w1.netbsd-base.*
make /home/build/pool/w1.netbsd-base
make[1]: Entering directory '/home/build/repo-under-test/testing/libvirt'
cp /home/build/pool/NetBSD-10.0_RC1-amd64.iso /home/build/pool/w1.netbsd-base.iso.tmp
sed -e 's;@@GATEWAY@@;192.168.234.1;' -e 's;@@BENCHDIR@@;/home/build/repo-under-test;' -e 's;@@LOCALDIR@@;/tmp/pool;' -e 's;@@POOLDIR@@;/home/build/pool;' -e 's;@@SOURCEDIR@@;/home/build/repo-under-test;' -e 's;@@TESTINGDIR@@;/home/build/repo-under-test/testing;' -e 's;@@USER@@;1000;' -e 's;@@GROUP@@;107;' -e 's;@@PREFIX@@;w1.;' \
	netbsd/base.sh \
	> /home/build/pool/w1.netbsd-base.sh
: this mangles file/directory names
growisofs -M /home/build/pool/w1.netbsd-base.iso.tmp -l \
	-input-charset utf-8 \
	-graft-points \
	/base.sh=/home/build/pool/w1.netbsd-base.sh
Executing 'mkisofs -C 16,304032 -M /dev/fd/3 -l -input-charset utf-8 -graft-points /base.sh=/home/build/pool/w1.netbsd-base.sh | builtin_dd of=/home/build/pool/w1.netbsd-base.iso.tmp obs=32k seek=19002'
xorriso 1.5.6 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Local character set is now assumed as: 'utf-8'
Drive current: -outdev 'stdio:/dev/fd/1'
Media current: stdio file, sequential
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 4096g free
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE :    9438 nodes read in 1 seconds
libisofs: WARNING : Found hidden El-Torito image. Its size could not be figured out, so image modify or boot image patching may lead to bad results.
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 20 * 2048 , 2048 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev 'stdio:/dev/fd/3'
Media current: stdio file, random read-only
Media status : is written , is closed
Boot record  : El Torito
Media summary: 1 session, 304019 data blocks,  594m data,     0 free
Volume id    : 'NETBSD_100_RC1'
Added to ISO image: file '/base.sh'='/home/build/pool/w1.netbsd-base.sh'
xorriso : UPDATE :       1 files added in 1 seconds
xorriso : UPDATE :       1 files added in 1 seconds
xorriso : NOTE : Discarded boot image from old session
ISO image produced: 3132 sectors
Written to medium : 3132 sectors at LBA 0
Writing to 'stdio:/dev/fd/1' completed successfully.

builtin_dd: 3136*2KB out @ average infx1352KBps
/home/build/pool/w1.netbsd-base.iso.tmp: copying volume descriptor(s)
mv /home/build/pool/w1.netbsd-base.iso.tmp /home/build/pool/w1.netbsd-base.iso
: @=/home/build/pool/w1.netbsd-base *=netbsd
: clean up old domains
./kvm-uninstall-domain.sh /home/build/pool/w1.netbsd-base
No domain w1.netbsd-base
: use script to drive build of new domain
./kvm-install-base.py \
	os "netbsd" \
	domain "w1.netbsd-base" \
	gateway 192.168.234.1 \
	benchdir /home/build/repo-under-test \
	pooldir /home/build/pool \
	-- \
	sudo virt-install \
		--connect=qemu:///system \
 --check=path_in_use=off \
 --graphics=none \
 --virt-type=kvm \
 --noreboot \
 --console=pty,target_type=serial \
 --cpu=host-passthrough \
 --network=network:swandefault,model=virtio \
 --rng=type=random,device=/dev/random \
 --security=type=static,model=dac,label='1000:107',relabel=yes \
		--vcpus=1 \
		--memory=2048 \
		--name=w1.netbsd-base \
		--os-variant=netbsd9.0 \
		--disk=path=/home/build/pool/w1.netbsd-base.qcow2,size=10,bus=virtio,format=qcow2 \
		--filesystem=target=pool,type=mount,accessmode=squash,source=/home/build/pool \
		--cdrom=/home/build/pool/NetBSD-10.0_RC1-amd64-boot.iso --disk=path=/home/build/pool/w1.netbsd-base.iso,readonly=on,device=cdrom
os netbsd
domain w1.netbsd-base
gateway 192.168.234.1
benchdir /home/build/repo-under-test
pooldir /home/build/pool
command ['sudo', 'virt-install', '--connect=qemu:///system', '--check=path_in_use=off', '--graphics=none', '--virt-type=kvm', '--noreboot', '--console=pty,target_type=serial', '--cpu=host-passthrough', '--network=network:swandefault,model=virtio', '--rng=type=random,device=/dev/random', '--security=type=static,model=dac,label=1000:107,relabel=yes', '--vcpus=1', '--memory=2048', '--name=w1.netbsd-base', '--os-variant=netbsd9.0', '--disk=path=/home/build/pool/w1.netbsd-base.qcow2,size=10,bus=virtio,format=qcow2', '--filesystem=target=pool,type=mount,accessmode=squash,source=/home/build/pool', '--cdrom=/home/build/pool/NetBSD-10.0_RC1-amd64-boot.iso', '--disk=path=/home/build/pool/w1.netbsd-base.iso,readonly=on,device=cdrom']
WARNING  CDROM media does not print to the text console by default, so you likely will not see text install output. You might want to use --location. See the man page for examples of using --location with CDROM media

Starting install...
Allocating 'w1.netbsd-base.qcow2'                           |    0 B  00:00 ... 
Creating domain...                                          |    0 B  00:00     
Running text console command: virsh --connect qemu:///system console w1.netbsd-base
Connected to domain 'w1.netbsd-base'
Escape character is ^] (Ctrl + ])
20 seconds.     
21287568+782888+1314264 [939219+1396200+1022961]=0x1b349b0
WARNING: couldn't open /var/db/entropy-file
Loading /stand/amd64/10.0/modules/cd9660/cd9660.kmod  
WARNING: 1 module failed to load
[   1.0000000] cpu_rng: rdrand/rdseed
[   1.0000000] entropy: ready
[   1.0000000] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
[   1.0000000]     2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013,
[   1.0000000]     2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023
[   1.0000000]     The NetBSD Foundation, Inc.  All rights reserved.
[   1.0000000] Copyright (c) 1982, 1986, 1989, 1991, 1993
[   1.0000000]     The Regents of the University of California.  All rights reserved.

[   1.0000000] NetBSD 10.0_RC1 (GENERIC) #0: Thu Dec 14 17:57:38 UTC 2023
[   1.0000000] 	mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC
[   1.0000000] total memory = 2047 MB
[   1.0000000] avail memory = 1953 MB
[   1.0000000] WARNING: module error: module `cd9660' pushed by boot loader already exists
[   1.0000040] mainbus0 (root)
[   1.0000040] ACPI: RSDP 0x00000000000F5210 000014 (v00 BOCHS )
[   1.0000040] ACPI: RSDT 0x000000007FFE1B29 000030 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[   1.0000040] ACPI: FACP 0x000000007FFE1A15 000074 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[   1.0000040] ACPI: DSDT 0x000000007FFE0040 0019D5 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[   1.0000040] ACPI: FACS 0x000000007FFE0000 000040
[   1.0000040] ACPI: APIC 0x000000007FFE1A89 000078 (v03 BOCHS  BXPC     00000001 BXPC 00000001)
[   1.0000040] ACPI: WAET 0x000000007FFE1B01 000028 (v01 BOCHS  BXPC     00000001 BXPC 00000001)
[   1.0000040] ACPI: 1 ACPI AML tables successfully acquired and loaded
[   1.0000040] ioapic0 at mainbus0 apid 0
[   1.0000040] cpu0 at mainbus0 apid 0
[   1.0000040] cpu0: Intel(R) Xeon(R) w3-2423, id 0x806f8
[   1.0000040] cpu0: node 0, package 0, core 0, smt 0
[   1.0000040] acpi0 at mainbus0: Intel ACPICA 20221020
[   1.0000040] acpi0: fixed power button present
[   1.0171120] qemufwcfg0 at acpi0 (FWCF, QEMU0002): io 0x510-0x51b
[   1.0171120] com0 at acpi0 (COM1, PNP0501-1): io 0x3f8-0x3ff irq 4
[   1.0171120] com0: ns16550a, 16-byte FIFO
[   1.0171120] com0: console
[   1.0171120] pckbc1 at acpi0 (KBD, PNP0303) (kbd port): io 0x60,0x64 irq 1
[   1.0171120] pckbc2 at acpi0 (MOU, PNP0F13) (aux port): irq 12
[   1.0171120] fdc0 at acpi0 (FDC0, PNP0700): io 0x3f2-0x3f5,0x3f7 irq 6 drq 2
[   1.0171120] ACPI: Enabled 2 GPEs in block 00 to 0F
[   1.0171120] pckbd0 at pckbc1 (kbd slot)
[   1.0171120] pckbc1: using irq 1 for kbd slot
[   1.0171120] wskbd0 at pckbd0 mux 1
[   1.0171120] pms0 at pckbc1 (aux slot)
[   1.0171120] pckbc1: using irq 12 for aux slot
[   1.0171120] wsmouse0 at pms0 mux 0
[   1.0171120] pci0 at mainbus0 bus 0: configuration mode 1
[   1.0171120] pchb0 at pci0 dev 0 function 0: Intel 82441FX (PMC) PCI and Memory Controller (rev. 0x02)
[   1.0171120] pcib0 at pci0 dev 1 function 0: Intel 82371SB (PIIX3) PCI-ISA Bridge (rev. 0x00)
[   1.0171120] piixide0 at pci0 dev 1 function 1: Intel 82371SB IDE Interface (PIIX3) (rev. 0x00)
[   1.0171120] piixide0: primary channel interrupting at ioapic0 pin 14
[   1.0171120] atabus0 at piixide0 channel 0
[   1.0171120] piixide0: secondary channel interrupting at ioapic0 pin 15
[   1.0171120] atabus1 at piixide0 channel 1
[   1.0171120] piixpm0 at pci0 dev 1 function 3: Intel 82371AB (PIIX4) Power Management Controller (rev. 0x03)
[   1.0171120] piixpm0: 24-bit timer
[   1.0171120] piixpm0: interrupting at ioapic0 pin 9
[   1.0171120] iic0 at piixpm0 port 0: I2C bus
[   1.0171120] virtio0 at pci0 dev 2 function 0
[   1.0171120] virtio0: 9P transport device (id 9, rev. 0x00)
[   1.0171120] virtio0: no matching child driver; not configured
[   1.0171120] virtio1 at pci0 dev 3 function 0
[   1.0171120] virtio1: network device (id 1, rev. 0x00)
[   1.0171120] vioif0 at virtio1: features: 0x31870020<EVENT_IDX,INDIRECT_DESC,NOTIFY_ON_EMPTY,CTRL_MAC,CTRL_RX,CTRL_VQ,STATUS,MAC>
[   1.0171120] vioif0: Ethernet address 52:54:00:43:dc:fc
[   1.0171120] virtio1: config interrupting at msix0 vec 0
[   1.0171120] virtio1: queues interrupting at msix0 vec 1
[   1.0171120] uhci0 at pci0 dev 4 function 0: Intel 82801I USB UHCI Controller (rev. 0x03)
[   1.0171120] uhci0: interrupting at ioapic0 pin 11
[   1.0171120] usb0 at uhci0: USB revision 1.0
[   1.0171120] uhci1 at pci0 dev 4 function 1: Intel 82801I USB UHCI Controller (rev. 0x03)
[   1.0171120] uhci1: interrupting at ioapic0 pin 10
[   1.0171120] usb1 at uhci1: USB revision 1.0
[   1.0171120] uhci2 at pci0 dev 4 function 2: Intel 82801I USB UHCI Controller (rev. 0x03)
[   1.0171120] uhci2: interrupting at ioapic0 pin 10
[   1.0171120] usb2 at uhci2: USB revision 1.0
[   1.0171120] ehci0 at pci0 dev 4 function 7: Intel 82801I USB EHCI Controller (rev. 0x03)
[   1.0171120] ehci0: interrupting at ioapic0 pin 11
[   1.0171120] ehci0: 3 companion controllers, 2 ports each: uhci0 uhci1 uhci2
[   1.0171120] ehci0: Using DMA subregion for control data structures
[   1.0171120] usb3 at ehci0: USB revision 2.0
[   1.0171120] virtio2 at pci0 dev 5 function 0
[   1.0171120] virtio2: block device (id 2, rev. 0x00)
[   1.0171120] ld0 at virtio2: features: 0x10000a54<INDIRECT_DESC,CONFIG_WCE,FLUSH,BLK_SIZE,GEOMETRY,SEG_MAX>
[   1.0171120] virtio2: config interrupting at msix1 vec 0
[   1.0171120] virtio2: queues interrupting at msix1 vec 1
[   1.0171120] ld0: 10240 MB, 16383 cyl, 16 head, 63 sec, 512 bytes/sect x 20971520 sectors
[   1.0171120] virtio3 at pci0 dev 6 function 0
[   1.0171120] virtio3: memory balloon device (id 5, rev. 0x00)
[   1.0171120] viomb0 at virtio3: features: 0x10000000<INDIRECT_DESC>
[   1.0171120] virtio3: interrupting at ioapic0 pin 10
[   1.0171120] virtio4 at pci0 dev 7 function 0
[   1.0171120] virtio4: entropy device (id 4, rev. 0x00)
[   1.0171120] viornd0 at virtio4: features: 0x10000000<INDIRECT_DESC>
[   1.0171120] virtio4: interrupting at ioapic0 pin 11
[   1.0171120] isa0 at pcib0
[   1.0171120] attimer0 at isa0 port 0x40-0x43
[   1.0171120] pcppi0 at isa0 port 0x61
[   1.0171120] spkr0 at pcppi0: PC Speaker
[   1.0171120] wsbell at spkr0 not configured
[   1.0171120] midi0 at pcppi0: PC speaker
[   1.0171120] sysbeep0 at pcppi0
[   1.0171120] attimer0: attached to pcppi0
[   1.0171120] acpicpu0 at cpu0: ACPI CPU
[   1.0171120] vmt0 at cpu0
[   1.0171120] vmware: open failed, eax=0xffffffff, ecx=0x1e, edx=0x5658
[   1.0171120] vmt0: failed to open backdoor RPC channel (TCLO protocol)
[   1.2077360] atapibus0 at atabus0: 2 targets
[   1.3711794] cd0 at atapibus0 drive 0: <QEMU DVD-ROM, QM00001, 2.5+> cdrom removable
[   1.3711794] cd1 at atapibus0 drive 1: <QEMU DVD-ROM, QM00002, 2.5+> cdrom removable
[   1.3711794] uvm_fault(0xffffffff8190fb40, 0xffffaf80b451e000, 2) -> e
[   1.3711794] fatal page fault in supervisor mode
[   1.3711794] trap type 6 code 0x2 rip 0xffffffff80e87a6c cs 0x8 rflags 0x10206 cr2 0xffffaf80b451e000 ilevel 0 rsp 0xffffffff81d4dcf8
[   1.3711794] curlwp 0xffffffff8188abc0 pid 0.0 lowest kstack 0xffffffff81d482c0
kernel: page fault trap, code=0
Stopped in pid 0.0 (system) at  netbsd:memset+0x2c:     repe stosq      %es:(%rd
i)
memset() at netbsd:memset+0x2c
lwp_create() at netbsd:lwp_create+0x283
fork1() at netbsd:fork1+0x429
main() at netbsd:main+0x43a
ds          1f
es          d200
fs          dd00
gs          6fa4
rdi         ffffaf80b451e000
rsi         2b00
rbp         ffffffff81d4dd40
rbx         ffffaf80b451d000
rdx         b451fb80
rcx         370
rax         0
r8          ffffffff80cb4463    start_init
r9          ffffaf80b451d080
r10         8000000001d9a063
r11         0
r12         0
--db_more--

[[So I've a VM with input detached waiting for someone to hit the space bar ...]]

virsh console --force w1.netbsd-base
Connected to domain 'w1.netbsd-base'
Escape character is ^] (Ctrl + ])
r13         ffff879ad1e98900
r14         ffffffff8188abc0    lwp0
r15         0
rip         ffffffff80e87a6c    memset+0x2c
cs          8
rflags      10206
rsp         ffffffff81d4dcf8
ss          10
netbsd:memset+0x2c:     repe stosq      %es:(%rdi)
db{0}> 
db{0}> bt
memset() at netbsd:memset+0x2c
lwp_create() at netbsd:lwp_create+0x283
fork1() at netbsd:fork1+0x429
main() at netbsd:main+0x43a

>How-To-Repeat:

>Fix: