Re: booting xen [was Re: serial console puzzle]

To: NetBSD/xen Discussion List <port-xen%NetBSD.org@localhost>
Subject: Re: booting xen [was Re: serial console puzzle]
From: Patrick Welche <prlw1%cam.ac.uk@localhost>
Date: Mon, 10 May 2021 11:10:35 +0100

On Fri, May 07, 2021 at 02:55:52PM -0700, Greg A. Woods wrote:
> At Fri, 7 May 2021 16:44:00 +0100, Patrick Welche <prlw1%cam.ac.uk@localhost> wrote:
> Subject: Re: booting xen [was Re: serial console puzzle]
> >
> > On Fri, May 07, 2021 at 08:52:02PM +0530, Mathew, Cherry G. wrote:
> > > On 2021, ???? 7 8:35:54 PM IST, Patrick Welche <prlw1%cam.ac.uk@localhost> wrote:
> > > >
> > > >[   1.0000000] NetBSD 9.99.82 (XEN3_DOM0) #4: Wed Apr 28 10:53:21 BST
> > > >2021
> > > >...
> > > >[   1.0000030] pci14: i/o space, memory space enabled
> > > >[   1.0000030] entropy: WARNING: extracting entropy too early
> > > >(XEN) mm.c:2980:d0v0 Bad type (saw e400000000000001 != exp
> > > >2000000000000000) fo)
> > > >(XEN) mm.c:1142:d0v0 Attempt to create linear p.t. with write perms
> > >
> > > uhuh. on a hunch:
> > >
> > > can you reduce the dom0 RAM to, say 1G, or play with the configuration (dom0_mem=1G) to see if has any effect ?
> >
> > I should have said, that output was from
> >
> > menu=Boot Xen:consdev com0,57600;load /netbsd-XEN3_DOM0 console=com0;multiboot /xen-debug.gz dom0_mem=1024M console=com1 com1=57600,8n1,0x3f8
> >
> > so exactly like an older message, without the rndseed bit, which apparently
> > caused the "not an ELF binary" error.
> >
> > (XEN) PHYSICAL MEMORY ARRANGEMENT:
> > (XEN)  Dom0 alloc.:   0000000618000000->000000061a000000 (253952 pages to be al)
> > (XEN) VIRTUAL MEMORY ARRANGEMENT:
> > (XEN)  Loaded kernel: ffffffff80200000->ffffffff81150ef8
> > (XEN)  Init. ramdisk: ffffffff81151000->ffffffff81151000
> > (XEN)  Phys-Mach map: ffffffff81151000->ffffffff81351000
> > (XEN)  Start info:    ffffffff81351000->ffffffff813514b8
> > (XEN)  Xenstore ring: 0000000000000000->0000000000000000
> > (XEN)  Console ring:  0000000000000000->0000000000000000
> > (XEN)  Page tables:   ffffffff81352000->ffffffff8135f000
> > (XEN)  Boot stack:    ffffffff8135f000->ffffffff81360000
> > (XEN)  TOTAL:         ffffffff80000000->ffffffff81400000
> > (XEN)  ENTRY ADDRESS: ffffffff8023d000
> >
> > [   1.0000000] NetBSD 9.99.82 (XEN3_DOM0) #4: Wed Apr 28 10:53:21 BST 2021
> > [   1.0000000]  prlw1@waxwing:/usr/src/sys/arch/amd64/compile/obj.amd64/XEN3_DO0
> > [   1.0000000] total memory = 1024 MB
> > [   1.0000000] avail memory = 971 MB
> 
> I take it that it (NetBSD now, not Xen) is still panic-ing though?

Yes

> And is Xen still printing the same "(Xen) mm.c:" messages too?

Yes:

[   1.0000030] entropy: WARNING: extracting entropy too early
(XEN) mm.c:2980:d0v0 Bad type (saw e400000000000001 != exp 2000000000000000) fo)
(XEN) mm.c:1142:d0v0 Attempt to create linear p.t. with write perms
[   1.0000030] xpq_flush_queue: 2 entries (0 successful) on cpu0 (0)
[   1.0000030] panic: HYPERVISOR_mmu_update failed, ret: -22

Further up I see

(XEN) d0: Forcing read-only access to MFN fed00

Is the d0 related to d0v0?

[   1.0000000] hypervisor0: features:  pae_pgdir_above_4gb mmu_pt_update_presers
(XEN) grant_table.c:1861:d0v0 Expanding d0 grant table from 1 to 2 frames
(XEN) grant_table.c:1861:d0v0 Expanding d0 grant table from 2 to 3 frames
...
(XEN) grant_table.c:1861:d0v0 Expanding d0 grant table from 62 to 63 frames
(XEN) grant_table.c:1861:d0v0 Expanding d0 grant table from 63 to 64 frames

?

> One other difference with what I do is that I've never tried to boot a
> gzip'ed version of any kernel (xen or netbsd) -- I just don't see the
> point and I always unpack them.  It shouldn't be the problem though as I
> believe others do exercise the decompression code in the boot loader
> regularly.

Just tried

menu=Greg orig:load /netbsd-XEN3_DOM0 -v bootdev=sd0a console=xencons;multiboot /xen-debug bootscrub=false dom0_mem=1024M console=com1 com1=57600,8n1,0x3f8

with a decompressed xen-debug, (/netbsd-XEN3_DOM0 was already decompressed)
but no change.

> Grasping at straws I don't know much about I'm also wondering if your
> system has some slightly weird memory layout too.  There's usually some
> way to tell the EFI shell to print info about memory layout -- Google
> suggests "memmap".  Beyond comparing the results to what Xen reports the
> real trick here though might be recognizing something wrong/different
> with what the firmware reports vs. the actual hardware.


May I just check the conclusion of the "first" thread: none of your working
xen systems boot using EFI? (It took a bit of doing to turn this working
EFI+gpt+zfs on /+GENERIC box into a BIOS+disklabel+attempt_xen box.)


What should I be looking for in the memory layout? With the "Greg orig"
menu option I see:

 Xen 4.15.0nb0
(XEN) Xen version 4.15.0nb0 (prlw1@) (gcc (nb1 20210411) 10.3.0) debug=y Thu Ap1
(XEN) Latest ChangeSet: 
(XEN) build-id: d2ee973db9f01886c1297a3a469888de162702c6
(XEN) Bootloader: NetBSD/x86 BIOS Boot, Revision 5.11 (Tue Apr 20 14:32:11 UTC )
(XEN) Command line: bootscrub=false dom0_mem=1024M console=com1 com1=57600,8n1,8
(XEN) Xen image load base address: 0
...
(XEN) CPU Vendor: AMD, Family 16 (0x10), Model 9 (0x9), Stepping 1 (raw 00100f9)
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 000000000009dfff] (usable)
(XEN)  [0000000000100000, 00000000df678fff] (usable)
(XEN)  [00000000df679000, 00000000df68efff] (reserved)
(XEN)  [00000000df68f000, 00000000df6cdfff] (ACPI data)
(XEN)  [00000000df6ce000, 00000000dfffffff] (reserved)
(XEN)  [00000000f0000000, 00000000f3ffffff] (reserved)
(XEN)  [00000000fe000000, 00000000fec8ffff] (reserved)
(XEN)  [00000000fec94000, 00000000feccffff] (reserved)
(XEN)  [00000000fecd4000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 000000181fffffff] (usable)
(XEN) New Xen image base address: 0xdf000000
...
(XEN) System RAM: 98294MB (100653148kB)
(XEN) SRAT: Node 0 PXM 0 0-a0000
(XEN) SRAT: Node 0 PXM 0 100000-e0000000
(XEN) SRAT: Node 0 PXM 0 100000000-620000000
(XEN) SRAT: Node 1 PXM 1 620000000-c20000000
(XEN) SRAT: Node 2 PXM 2 c20000000-1220000000
(XEN) SRAT: Node 3 PXM 3 1220000000-1820000000
...
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features:
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk LFENCE, SPEC_CTRL: No, Other: BRANCH_HARDEN
(XEN)   Support for HVM VMs: RSB
(XEN)   Support for PV VMs: RSB
(XEN)   XPTI (64-bit PV only): Dom0 disabled, DomU disabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
...
(XEN) Freed 1024kB unused BSS memory
(XEN) alt table ffff82d040489430 -> ffff82d04049729a
(XEN) AMD-Vi: Disabled HAP memory map sharing with IOMMU
...
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) alt table ffff82d040489430 -> ffff82d04049729a
...
(XEN) ELF: phdr: paddr=0xffffffff80200000 memsz=0xe07000
(XEN) ELF: memory: 0xffffffff80200000 -> 0xffffffff81007000
(XEN) ELF: note: GUEST_OS = "NetBSD"
(XEN) ELF: note: GUEST_VERSION = "4.99"
(XEN) ELF: note: XEN_VERSION = "xen-3.0"
(XEN) ELF: note: VIRT_BASE = 0xffffffff80000000
(XEN) ELF: note: PADDR_OFFSET = 0xffffffff80000000
(XEN) ELF: note: ENTRY = 0xffffffff8023d000
(XEN) ELF: note: HYPERCALL_PAGE = 0xffffffff8023e000
(XEN) ELF: note: HV_START_LOW = 0xffff800000000000
(XEN) ELF: note: FEATURES = "writable_descriptor_tables|auto_translated_physmap"
(XEN) ELF: note: PAE_MODE = "yes"
(XEN) ELF: note: unknown (0xd)
(XEN) ELF: note: LOADER = "generic"
(XEN) ELF: note: SUSPEND_CANCEL = 0
(XEN) ELF: note: BSD_SYMTAB = "yes"
(XEN) ELF: using notes from SHT_NOTE section
(XEN) ELF: addresses:
(XEN)     virt_base        = 0xffffffff80000000
(XEN)     elf_paddr_offset = 0xffffffff80000000
(XEN)     virt_offset      = 0x0
(XEN)     virt_kstart      = 0xffffffff80200000
(XEN)     virt_kend        = 0xffffffff81150ef8
(XEN)     virt_entry       = 0xffffffff8023d000
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0xffffffff80200000 -> 0xffffffff8100
(XEN)  Dom0 symbol map 0xffffffff81007000 -> 0xffffffff81150ef8
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000618000000->000000061a000000 (253952 pages to be al)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff81150ef8
(XEN)  Init. ramdisk: ffffffff81151000->ffffffff81151000
(XEN)  Phys-Mach map: ffffffff81151000->ffffffff81351000
(XEN)  Start info:    ffffffff81351000->ffffffff813514b8
(XEN)  Xenstore ring: 0000000000000000->0000000000000000
(XEN)  Console ring:  0000000000000000->0000000000000000
(XEN)  Page tables:   ffffffff81352000->ffffffff8135f000
(XEN)  Boot stack:    ffffffff8135f000->ffffffff81360000
(XEN)  TOTAL:         ffffffff80000000->ffffffff81400000
(XEN)  ENTRY ADDRESS: ffffffff8023d000
(XEN) Dom0 has maximum 16 VCPUs
(XEN) ELF: phdr 0 at 0xffffffff80200000 -> 0xffffffff80e93a30
(XEN) Initial low memory virq threshold set at 0x4000 pages.
...
[   1.0000000] NetBSD 9.99.82 (XEN3_DOM0) #4: Wed Apr 28 10:53:21 BST 2021
[   1.0000000]  prlw1@waxwing:/usr/src/sys/arch/amd64/compile/obj.amd64/XEN3_DO0
[   1.0000000] total memory = 1024 MB
[   1.0000000] avail memory = 971 MB
...
[   1.0000030] entropy: WARNING: extracting entropy too early
(XEN) mm.c:2980:d0v0 Bad type (saw e400000000000001 != exp 2000000000000000) fo)
(XEN) mm.c:1142:d0v0 Attempt to create linear p.t. with write perms
[   1.0000030] xpq_flush_queue: 2 entries (0 successful) on cpu0 (0)
[   1.0000030] panic: HYPERVISOR_mmu_update failed, ret: o22
[   1.0000030] cpu0: Begin traceback...
[   1.0000030] vpanic() at netbsd:vpanic+0x14a
[   1.0000030] device_printf() at netbsd:device_printf
[   1.0000030] xpq_queue_machphys_update() at netbsd:xpq_queue_machphys_update
[   1.0000030] pmap_zero_page() at netbsd:pmap_zero_page+0xe3
[   1.0000030] uvm_pagealloc_strat() at netbsd:uvm_pagealloc_strat+0x1ef
[   1.0000030] pmap_get_physpage() at netbsd:pmap_get_physpage+0x1cb
[   1.0000030] pmap_growkernel() at netbsd:pmap_growkernel+0x1b3
[   1.0000030] uvm_map_prepare() at netbsd:uvm_map_prepare+0x3a2
[   1.0000030] uvm_map() at netbsd:uvm_map+0x70
[   1.0000030] ubc_init() at netbsd:ubc_init+0x15b
[   1.0000030] main() at netbsd:main+0x33b
[   1.0000030] cpu0: End traceback...
[   1.0000030] fatal breakpoint trap in supervisor mode
[   1.0000030] trap type 1 code 0 rip 0xffffffff8024093d cs 0xe030 rflags 0x2020
[   1.0000030] curlwp 0xffffffff80e75040 pid 0.0 lowest kstack 0xffffffff8138720



Other complaints in the log are

(XEN) Brought up 16 CPUs
...
(XEN) mtrr: your CPUs had inconsistent fixed MTRR settings
(XEN) mtrr: probably your BIOS does not setup all CPUs.
(XEN) mtrr: corrected configuration.

(XEN) io_apic.c:2386: IO-APIC: apic=0, pin=0, irq=0
(XEN) IO-APIC: new_entry=000100f0
(XEN) IO-APIC: old_entry=00010000 pirq=-1
(XEN) IO-APIC: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) io_apic.c:2386: IO-APIC: apic=0, pin=2, irq=0
(XEN) IO-APIC: new_entry=000100f0
(XEN) IO-APIC: old_entry=000000f0 pirq=-1
(XEN) IO-APIC: Attempt to modify IO-APIC pin for in-use IRQ!
(XEN) io_apic.c:2386: IO-APIC: apic=0, pin=4, irq=4
(XEN) IO-APIC: new_entry=000100f1
(XEN) IO-APIC: old_entry=000000f1 pirq=-1
(XEN) IO-APIC: Attempt to modify IO-APIC pin for in-use IRQ!

[   1.0000000] ioapic1 at mainbus0 apid 1: pa 0xfec80000, version 0x21, 32 pins
[   1.0000000] ioapic1: can't remap to apid 1
[   1.0000000] ioapic2 at mainbus0 apid 2: pa 0xfecc0000, version 0x21, 32 pins
[   1.0000000] ioapic2: can't remap to apid 2



Cheers,

Patrick

Follow-Ups:
- Re: booting xen [was Re: serial console puzzle]
  - From: Brian Buhrow
- Re: booting xen [was Re: serial console puzzle]
  - From: Brian Buhrow

References:
- Re: booting xen [was Re: serial console puzzle]
  - From: Greg A. Woods
- Re: booting xen [was Re: serial console puzzle]
  - From: Patrick Welche
- Re: booting xen [was Re: serial console puzzle]
  - From: Mathew, Cherry G.
- Re: booting xen [was Re: serial console puzzle]
  - From: Patrick Welche
- Re: booting xen [was Re: serial console puzzle]
  - From: Greg A. Woods

Prev by Date: Re: booting xen [was Re: serial console puzzle]
Next by Date: Problems booting netbsd DOM0
Previous by Thread: Re: booting xen [was Re: serial console puzzle]
Next by Thread: Re: booting xen [was Re: serial console puzzle]
Indexes:

Home | Main Index | Thread Index | Old Index