Re: NetBSD/i386 5.99.64 locks up while running "/etc/daily"

[Matthias Scheler <tron%zhadum.org.uk@localhost>]

On Sat, Feb 04, 2012 at 10:01:56PM +0000, Matthias Scheler wrote:
> my NetBSD/i386 5.99.64 virtual machine (VMware Fusion 4.1.1, 6 virtual cores)
> locks up reproducably while running "/etc/daily". This is what "top" says
> when it happens:
> 
> load averages:  0.00,  0.01,  0.02;               up 0+00:15:47        
> 21:55:00
> 40 processes: 38 sleeping, 2 on CPU
> CPU0 states:  0.0% user,  0.0% nice, 24.6% system,  0.0% interrupt, 75.4% idle
> CPU1 states:  0.0% user,  0.0% nice, 16.6% system,  0.0% interrupt, 83.4% idle
> CPU2 states:  0.0% user,  0.0% nice, 11.0% system,  0.0% interrupt, 89.0% idle
> CPU3 states:  0.0% user,  0.0% nice, 55.5% system,  0.0% interrupt, 44.5% idle
> CPU4 states:  0.0% user,  0.0% nice,  4.6% system,  0.0% interrupt, 95.4% idle
> CPU5 states:  0.0% user,  0.0% nice, 12.4% system,  0.0% interrupt, 87.6% idle
> Memory: 20M Act, 13M Inact, 13M Wired, 8384K Exec, 8000K File, 2547M Free
> Swap: 4096M Total, 4096M Free
> 
>   PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
>     0 root       0    0     0K   14M CPU/4     15:11  0.00%   125% [system]
>   893 root      85    0  4172K 2184K vmem/0     0:17  0.00%  0.00% find
>   931 root      43    0  3364K 1772K CPU/5      0:00  0.00%  0.00% top
>   228 root      85    0  5344K 7396K select/0   0:00  0.00%  0.00% amd
>   405 root      85    0  5920K 6092K pause/5    0:00  0.00%  0.00% ntpd
>   290 root      85    0    10M 4588K select/5   0:00  0.00%  0.00% sshd
> 
> I've managed to enter "ddb" but the current process isn't doing anything
> interesting. I've tried "machine cpu 4" followed by "bt" but CPU 4
> is only in the idle loop.
> 
> Any ideas what to try next? This virtual machine worked fine with
> NetBSD/i386 5.99.60.

Juergen Hannken-Illjes suggested to use "top -t". Here is the output:

load averages:  0.02,  0.07,  0.04;               up 0+00:05:51                 
                                               14:33:29
116 threads: 26 idle, 1 runnable, 83 sleeping, 6 on CPU
CPU0 states:  0.0% user,  0.0% nice, 28.3% system,  0.0% interrupt, 71.7% idle
CPU1 states:  0.0% user,  0.0% nice, 17.8% system,  0.0% interrupt, 82.2% idle
CPU2 states:  0.0% user,  0.0% nice,  6.6% system,  0.0% interrupt, 93.4% idle
CPU3 states:  0.0% user,  0.0% nice, 59.9% system,  0.0% interrupt, 40.1% idle
CPU4 states:  0.0% user,  0.0% nice,  5.0% system,  0.0% interrupt, 95.0% idle
CPU5 states:  0.0% user,  0.0% nice, 14.6% system,  0.0% interrupt, 85.4% idle
Memory: 23M Act, 12M Inact, 13M Wired, 8368K Exec, 8092K File, 2542M Free
Swap: 4096M Total, 4096M Free

  PID   LID USERNAME PRI STATE      TIME   WCPU    CPU NAME      COMMAND
    0    70 root     126 CPU/3      1:50 51.71% 51.71% pgdaemon  [system]
    0     7 root     127 RUN/0      0:55 25.59% 25.59% xcall/0   [system]
    0    46 root     127 xcall/5    0:07 17.43% 17.43% xcall/5   [system]
    0    22 root     127 xcall/1    0:36 14.01% 14.01% xcall/1   [system]
    0    28 root     127 xcall/2    0:12  4.54%  4.54% xcall/2   [system]
    0    34 root     127 xcall/3    0:12  2.78%  2.78% xcall/3   [system]
    0    40 root     127 xcall/4    0:07  2.05%  2.05% xcall/4   [system]
  829     1 root      84 vmem/0     0:18  0.00%  0.00% -         find
  290     1 root      43 CPU/1      0:00  0.00%  0.00% -         top
    0    35 root       0 CPU/4      0:00  0.00%  0.00% idle/4    [system]
    0    23 root       0 CPU/2      0:00  0.00%  0.00% idle/2    [system]
    0     2 root       0 CPU/0      0:00  0.00%  0.00% idle/0    [system]
    0    41 root       0 CPU/5      0:00  0.00%  0.00% idle/5    [system]
    0    11 root     125 cacheg/4   0:00  0.00%  0.00% cachegc   [system]
    0    72 root     125 aiodon/0   0:00  0.00%  0.00% aiodoned  [system]
    0    61 root     125 vmem_r/0   0:00  0.00%  0.00% vmem_reha [system]
    0     1 root     125 uvm/0      0:00  0.00%  0.00% swapper   [system]
    0     8 root     125 mod_un/0   0:00  0.00%  0.00% modunload [system]
    0     9 root     125 vdrain/4   0:00  0.00%  0.00% vdrain    [system]
    0    10 root     125 vrele/4    0:00  0.00%  0.00% vrele     [system]
    0    71 root     124 syncer/1   0:00  0.00%  0.00% ioflush   [system]
    0    73 root     123 physio/5   0:00  0.00%  0.00% physiod   [system]
    0    63 root      96 usbevt/5   0:00  0.00%  0.00% usb0      [system]
    0    16 root      96 smtask/0   0:00  0.00%  0.00% sysmon    [system]
    0    64 root      96 usbtsk/5   0:00  0.00%  0.00% usbtask-h [system]
    0    65 root      96 usbtsk/5   0:00  0.00%  0.00% usbtask-d [system]
    0    75 root      96 nfsiod/1   0:00  0.00%  0.00% nfsio     [system]
    0    66 root      96 usbevt/5   0:00  0.00%  0.00% usb1      [system]
    0    67 root      96 crypto/5   0:00  0.00%  0.00% cryptoret [system]
    0    76 root      96 nfsiod/1   0:00  0.00%  0.00% nfsio     [system]
    0    77 root      96 nfsiod/1   0:00  0.00%  0.00% nfsio     [system]
    0    12 root      96 nfssil/0   0:00  0.00%  0.00% nfssilly  [system]
    0    74 root      96 nfsiod/1   0:00  0.00%  0.00% nfsio     [system]
    0    13 root      96 sopend/0   0:00  0.00%  0.00% sopendfre [system]
    0    14 root      96 pmfeve/0   0:00  0.00%  0.00% pmfevent  [system]
    0    62 root      96 unpgc/1    0:00  0.00%  0.00% unpgc     [system]
    0    15 root      96 pmfsus/0   0:00  0.00%  0.00% pmfsuspen [system]
    0    60 root      96 sccomp/0   0:00  0.00%  0.00% atapibus0 [system]
    0    50 root      96 sccomp/0   0:00  0.00%  0.00% scsibus0  [system]
    0    48 root      96 atath/0    0:00  0.00%  0.00% atabus0   [system]
    0    49 root      96 atath/0    0:00  0.00%  0.00% atabus1   [system]
    0    47 root      96 pmsres/0   0:00  0.00%  0.00% pms0      [system]
  629     1 tron      85 ttyraw/3   0:00  0.00%  0.00% -         zsh

There seems to be some kind of CPU cross call storm happening.

        Kind regards

-- 
Matthias Scheler                                  http://zhadum.org.uk/