Re: kern/54818: 9.0_RC1 pagedaemon spins

[Lars Reichardt <lars%paradoxon.info@localhost>]

On 05.03.2020 16:41, Havard Eidnes wrote:
> > >    buf16k 16384 1411 0 1270 241 205 36 93 1 1 0
> > >    buf1k 1024 2 0 2 1 0 1 1 1 1 1
> > >    buf2k 2048 9 0 9 5 4 1 5 1 1 1
> > >    buf32k 32768 223292 0 193503 92993 73624 19369 37065 1 1 0
> > >    buf4k 4096 491370 0 391560 491371 391560 99811 179873 1 1 1
> > >    buf64k 65536 4 0 0 5 0 5 5 1 1 1
> > >    buf8k 8192 1865 0 1613 160 128 32 63 1 1 0
> > >    bufpl 288 210502 0 80506 15026 0 15026 15026 0 inf 111
> > > > > > these are very interesting:
> > These are the quantum caches for allocation virtual address space.
> >
> > No 4k allocation as the direct map is used (that's expected) and most
> > pools have a pool page size of 4k
> > but a lot of 64k allocations with the backing pool page size 256k.
> >
> > That is 64*63924 4091136kb worse of allocations
> > (15981 pool pages each 256k)
> >   and no releases at all seems like some leak to me.
> >
> > Does that happen when starting X?
> No.  It typically happens after a few days running.
>
> > Seems to be an intel drmkms judged from the list of pools.
> Correct:
>
> i915drmkms0 at pci0 dev 2 function 0: vendor 8086 product 0412 (rev. 0x06)
> drm: Memory usable by graphics device = 2048M
> drm: Supports vblank timestamp caching Rev 2 (21.10.2013).
> drm: Driver supports precise vblank timestamp query.
> i915drmkms0: interrupting at ioapic0 pin 16 (i915)
> intelfb0 at i915drmkms0
> i915drmkms0: info: registered panic notifier
> i915drmkms0: More than 8 outputs detected via ACPI
> intelfb0: framebuffer at 0xffff80013bc7d000, size 1920x1200, depth 32, stride 7680
> wsdisplay0 at intelfb0 kbdmux 1: console (default, vt100 emulation)
> wsmux1: connecting to wsdisplay0
>
> > The kmem arena is most likely a bit more than this mentioned 4g as the
> > machine seems to have 16gb?
> Physical ram is 16GB, yes.
>
> > It should be the second entry of the output of "pmap 0".
> Currently the first three lines from "pmap 0" is
>
> FFFF800000000000 473672K read/write/exec     [ anon ]
> FFFF80001CE92000 4166220K read/write/exec     [ anon ]
> FFFF80011B325000 524288K read/write/exec     [ pager_map ]
>
> > >    kva-12288 12288 35 0 0 2 0 2 2 0 inf 0
> > >    kva-16384 16384 17 0 0 2 0 2 2 0 inf 0
> > >    kva-20480 20480 84 0 0 7 0 7 7 0 inf 0
> > >    kva-24576 24576 9 0 0 1 0 1 1 0 inf 0
> > >    kva-28672 28672 3 0 0 1 0 1 1 0 inf 0
> > >    kva-32768 32768 1 0 0 1 0 1 1 0 inf 0
> > >    kva-36864 36864 3 0 0 1 0 1 1 0 inf 0
> > >    kva-40960 40960 108 0 0 18 0 18 18 0 inf 0
> > >    kva-49152 49152 1 0 0 1 0 1 1 0 inf 0
> > >    kva-65536 65536 63924 0 0 15981 0 15981 15981 0 inf 0
> > >    kva-8192 8192 52 0 0 2 0 2 2 0 inf 0
> > ...
> > I'm not aware of any pool that allocates from the 64k quantum cache so
> > it doesn't surprise me that that pagedaemon/pool_drain
> > isn't able to free anything.
> Hm.  I know too little about the mechanism, but having something
> allocate from it and nothing to release when there's pressure
> looks like a recipe for disaster.
Those kva pools are caches that cache some virtual address space 
allocated from the kmem arena no memory is directly allocated for them.

See: https://nxr.netbsd.org/xref/src/sys/uvm/uvm_km.c#783

uvm_km_kmem_alloc is use by most pools allocators to get memory for 
their pool_pages.

vmstat -mvW does show non used pools as well as their pool_pagesize.

As the only pools with 64k pool_pagesize are the buf pools and they have 
a different pool_allocator not allocation from kmem but from the kernel_map.

As far as I can guess currently the most likely candidate is kmem_alloc 
with a size of >48k and <= 64k as we only have kmem caches up to 
page_size and large allocations are done directly via uvm_km_kmem_alloc.


Monitoring those large allocation via dtrace should show us the consumer 
that doesn't free it's memory.

I'll try to come up a dtrace script for that, feel free to beat me on 
that my time is a bit rare at the moment.


he@: Yes, the report of NetBSD 8.1/i386 looks very much the same.

There is no direct map so the kva-4096 is used for all the pool_sized 
allocations but a lot of 64k allocations with almost no releases again.


Regards,

Lars


--
You will continue to suffer
if you have an emotional reaction to everything that is said to you.
True power is sitting back and observing everything with logic.
If words control you that means everyone else can control you.
Breathe and allow things to pass.

--- Bruce Lee