Re: domU freeze

[Emmanuel Dreyfus <manu%netbsd.org@localhost>]

On Thu, Oct 07, 2021 at 01:02:16PM +0000, Emmanuel Dreyfus wrote:
> I have the case of a domU that hangs every few hours. I am looking for
> hints about how to investigate it. 

Now I can use expect to run ddb commands when I detect the domU froze
(trick in http://mail-index.netbsd.org/port-xen/2021/10/08/msg010192.html)

Here is ps output on last freeze, detected because the domU stopped 
answering pings. I do not see anything shocking there. Another idea of
what to look at?

PID    LID S CPU     FLAGS       STRUCT LWP *               NAME WAIT
1526     1 3   0        80           c51a9040                ssh select
3953     1 3   0        80           c6191120                ssh select
9637     1 3   0        80           c509a480              httpd semwait
19120    1 3   0        80           c51f9620              httpd semwait
4467     1 3   0        80           c5139220              httpd semwait
11803    1 3   0        80           c509acc0              httpd semwait
16303    1 3   0        80           c51f9ba0              httpd kqueue
14537    1 3   0        80           c62216e0              httpd semwait
781      1 3   0        80           c604c100              httpd semwait
28812    1 3   0        80           c4c89c80              httpd semwait
10297    1 3   0        80           c51895a0              httpd semwait
10199    1 3   0        80           c61dd6c0              httpd semwait
12148    1 3   0        80           c516b7c0              httpd semwait
3941     1 3   0        80           c516b240              httpd semwait
15980    1 3   0        80           c5251900              httpd semwait
14525    1 3   0        80           c509aa00              httpd semwait
15509    1 3   0        80           c51e1080              httpd semwait
17360    1 3   0        80           c68231c0              httpd semwait
9619     1 3   0        80           c60433a0              httpd semwait
7139     1 3   0        80           c6823a00              httpd semwait
12169    1 3   1        80           c6191960              httpd semwait
15618    1 3   0        80           c51e18c0              httpd semwait
28462    2 5   1         0           c50031a0              httpd
28462    1 3   0        80           c624dc80              httpd semwait
5272     1 3   1        80           c6823740              local kqueue
10780    1 3   0        80           c604c3c0    trivial-rewrite kqueue
12199    1 3   1        80           c51c95e0            cleanup kqueue
6891     2 3   0        80           c47a5400             expect select
6891     1 3   0        80           c51394e0             expect parked
14389    2 3   0        80           c624d9c0             expect select
14389    1 3   1        80           c671d9e0             expect parked
9286     1 3   1        80           c6221420               cron pipe_rd
13799    1 3   0        80           c51e1600               cron pipe_rd
7118     1 3   1        80           c52510c0             pickup kqueue
1206 >   1 7   1         0           c43510c0              getty
1019     1 3   0        80           c5139d20              httpd select
1241     1 3   0        80           c50dca20       avahi-daemon select
1115     1 3   1        80           c50dc1e0               cron nanoslp
1193     1 3   1        80           c51a9880        dbus-daemon select
1042     1 3   0        80           c50f8a40              inetd kqueue
1059     1 3   1        80           c4373c20               qmgr kqueue
1031     1 3   1        80           c4c899c0             master kqueue
1508     1 3   0        80           c4c89440               sshd select
1103     1 3   0        80           c51c9320             powerd kqueue
672      1 3   0        80           c51e1340               ntpd pause
368      4 3   1        80           c4a62160          isc-timer parked
368      3 3   0        80           c47a56c0             sock-0 kqueue
368      2 3   0        80           c47a5140     isc-worker0000 parked
368      1 3   0        80           c50039e0              dhcpd parked
276      4 3   0        80           c4a629a0             sock-0 kqueue
276      3 3   1        80           c47a5980          isc-timer parked
276      2 3   0        80           c4a62c60     isc-worker0000 parked
276      1 3   0        80           c4c89700              named sigwait
247      1 3   1        80           c4c89180            syslogd kqueue
1        1 3   1        80           c4365c00               init wait
0       50 3   0       200           c4350ba0            physiod physiod
0       49 3   0       200           c4350360          pooldrain pooldrain
0       48 3   0       200           c4350620            ioflush syncer
0       47 3   0       200           c43508e0           pgdaemon pgdaemon
0       44 3   0       200           c434cb80           aiodoned aiodoned
0       43 3   1       200           c4351380        xen_balloon xen_balloon
0       42 3   1       200           c4373120             npfgc0 npfgcw
0       41 3   1       200           c4365940            rt_free rt_free
0       40 3   1       200           c4365680              unpgc unpgc
0       39 3   0       200           c43653c0    key_timehandler key_timehandler

0       38 3   1       200           c4365100    icmp6_wqinput/1 icmp6_wqinput
0       37 3   0       200           c435bbe0    icmp6_wqinput/0 icmp6_wqinput
0       36 3   1       200           c435b920          nd6_timer nd6_timer
0       35 3   1       200           c435b660     icmp_wqinput/1 icmp_wqinput
0       34 3   0       200           c435b3a0     icmp_wqinput/0 icmp_wqinput
0       33 3   0       200           c435b0e0           rt_timer rt_timer
0       32 3   0       200           c4351640        vmem_rehash vmem_rehash
0       31 3   0       200           c4351bc0             xenbus rdst
0       30 3   0       200           c4351900           xenwatch evtsq
0       20 3   1       200           c434c8c0            xcall/1 xcall
0       19 1   1       200           c434c600          softser/1
0       18 1   1       200           c434c340          softclk/1
0       17 1   1       200           c434c080          softbio/1
0       16 1   1       200           c433ab60          softnet/1
0       15 1   1       201           c433a8a0             idle/1
0       14 3   0       200           c433a5e0         pmfsuspend pmfsuspend
0       13 3   0       200           c433a320           pmfevent pmfevent
0       12 3   0       200           c433a060         sopendfree sopendfr
0       11 3   0       200           c3f68b40           nfssilly nfssilly
0       10 3   0       200           c3f68880            cachegc cachegc
0        9 3   1       200           c3f685c0             vdrain vdrain
0        8 3   0       200           c3f68300          modunload mod_unld
0        7 3   0       200           c3f68040            xcall/0 xcall
0        6 1   0       200           c3f66b20          softser/0
0        5 1   0       200           c3f66860          softclk/0
0        4 1   0       200           c3f665a0          softbio/0
0        3 1   0       200           c3f662e0          softnet/0
0    >   2 7   0       201           c3f66020             idle/0
0        1 3   0       200           c05c2320            swapper uvm


-- 
Emmanuel Dreyfus
manu%netbsd.org@localhost