Re: port-arm/56842

[Jim Spath <jspath55%gmail.com@localhost>]

The following reply was made to PR port-arm/56842; it has been noted by GNATS.

From: Jim Spath <jspath55%gmail.com@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: port-arm/56842
Date: Fri, 20 May 2022 09:20:51 -0400

 --000000000000aaee7005df715c67
 Content-Type: text/plain; charset="UTF-8"
 Content-Transfer-Encoding: quoted-printable
 
 Dave:
 
 Thank you for the feedback. I have NetBSD running also on a Pi3 and a Pi4;
 this is the first time getting a Zero 2W working. The other systems are
 running current:
 
 NetBSD [pi3] 9.99.82 NetBSD 9.99.82 (GENERIC64) #0: Tue Apr 27 05:40:29 UTC
 2021 mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/evbarm/compile/GENERIC64
 evbarm
 
 NetBSD [pi4] 9.99.93 NetBSD 9.99.93 (GENERIC64) #0: Sun Jan 2 23:46:21 UTC
 2022 mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/evbarm/compile/GENERIC64
 evbarm
 
 Neither of those, nor an earlier 9.2 system have shown cron hangs; I have
 the identical script running.
 
 Your questions:
 
 - what does ps(1) tell you about the state of cron?
 
 Nothing useful to me yet; but see below for results from top.
 
 
 USER      PID %CPU %MEM   VSZ   RSS TTY   STAT STARTED    TIME COMMAND
 
 root    18233  0.0  0.4  6728  1824 ?     Ss   Mon05PM 0:00.00
 /usr/sbin/cron
 
 - You might try enabling extra debugging information with the -x option.
 
 I tried one iteration with debug flags and captured logs but saw nothing
 useful there.
 
 - What happens if you disable particular cron entries, like the script
 meant to run every minute?
 
 I will try lowering the frequency, after doing a reboot and seeing if/when
 the issue recurs. It seems this might be a =E2=80=9Cslow leak=E2=80=9D that=
  will take
 patience to track.
 
 I investigated further and found hangs on both top and vmstat, at varying
 times.
 
 For vmstat, the first line (summary) is returned, but then nothing:
 
 n0b:jim> date ; vmstat 1 10
 
 Tue May 17 13:09:14 UTC 2022
 
 procs memory page disks faults cpu
 
 r b avm fre flt re pi po fr sr l0 n0 in sy cs us sy id
 
 1 0 304608 88784 23 0 0 0 0 0 0 0 8882 44 14 0 1 99
 
 ^C
 
 n0b:jim> date
 
 Tue May 17 13:09:45 UTC 2022
 
 That stall is inconsistent though, as the results today are nominal:
 
 n0b:jim> date
 
 Fri May 20 12:58:26 UTC 2022
 
 n0b:jim> vmstat 1 3
 
 procs memory page disks faults cpu
 
 r b avm fre flt re pi po fr sr l0 n0 in sy cs us sy id
 
 1 0 310320 82568 22 0 0 0 0 0 0 0 8870 43 13 0 1 99
 
 0 0 310320 82568 0 0 0 0 0 0 0 0 8826 32 11 0 1 99
 
 0 0 310320 82568 0 0 0 0 0 0 0 0 8902 30 10 0 1 99
 
 n0b:jim> date
 
 Fri May 20 12:58:36 UTC 2022
 
 n0b:jim>
 
 The top command starts up, displays some data, but then does not refresh.
 The data are incomplete (values are all 0):
 
 load averages: 0.01, 0.02, 0.00; up 11+21:37:48 13:06:53
 
 46 processes: 44 sleeping, 2 on CPU
 
 CPU0 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
 
 CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
 
 CPU2 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
 
 CPU3 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
 
 Memory: 298M Act, 104K Inact, 12M Wired, 15M Exec, 259M File, 86M Free
 
 Swap:
 
 
 Like vmstat, top worked later (except one core shows all zeroes).
 
 load averages: 0.01, 0.02, 0.00; up 14+21:32:34 13:01:39
 
 50 processes: 48 sleeping, 2 on CPU
 
 CPU0 states: 0.0% user, 0.0% nice, 0.0% system, 1.6% interrupt, 98.4% idle
 
 CPU1 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
 
 CPU2 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
 
 CPU3 states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
 
 Memory: 303M Act, 96K Inact, 12M Wired, 15M Exec, 262M File, 80M Free
 
 Swap:
 
 However, cron commands have not run since.
 
 My next steps will be:
 
 1. Reboot, taking note of initial state
 
 2. Try adding a swap device (have seen some odd Pi behavior with 0 swap)
 
 3. Decrease the cron job frequency
 
 
 Jim
 
 --000000000000aaee7005df715c67
 Content-Type: text/html; charset="UTF-8"
 Content-Transfer-Encoding: quoted-printable
 
 <div dir=3D"ltr">
 =09
 =09
 
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Dave=
 :</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Than=
 k you for the
 feedback. I have NetBSD running also on a Pi3 and a Pi4; this is the
 first time getting a Zero 2W working. The other systems are running
 current:</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">NetBSD [pi3] 9.99.82
 NetBSD 9.99.82 (GENERIC64) #0: Tue Apr 27 05:40:29 UTC 2021=20
 mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/evbarm/compile/GENERIC64
 evbarm</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">NetBSD [pi4] 9.99.93
 NetBSD 9.99.93 (GENERIC64) #0: Sun Jan  2 23:46:21 UTC 2022=20
 mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/evbarm/compile/GENERIC64
 evbarm</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">Neither of those,
 nor an earlier 9.2 system have shown cron hangs; I have the identical scrip=
 t running.</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">Your questions:</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">- </span><span style=3D"background=
 -color:transparent;font-variant-numeric:normal;font-variant-east-asian:norm=
 al"><font color=3D"#222222"><font face=3D"Arial, Helvetica, sans-serif"><fo=
 nt style=3D"font-size:12pt">what
 does ps(1) tell you about the state of cron?</font></font></font></span><br=
 ></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">Nothing useful to me yet;
 but see below for results from top.</span><br></p><p style=3D"line-height:1=
 00%;margin-bottom:0in;background:transparent"><span style=3D"background-col=
 or:transparent"><br></span></p><p style=3D"line-height:100%;margin-bottom:0=
 in;background:transparent">USER =C2=A0 =C2=A0 =C2=A0PID %CPU %MEM =C2=A0 VS=
 Z =C2=A0 RSS TTY =C2=A0 STAT STARTED =C2=A0 =C2=A0TIME COMMAND<br></p><p st=
 yle=3D"line-height:100%;margin-bottom:0in;background:transparent">root =C2=
 =A0 =C2=A018233 =C2=A00.0 =C2=A00.4 =C2=A06728 =C2=A01824 ? =C2=A0 =C2=A0 S=
 s =C2=A0 Mon05PM 0:00.00 /usr/sbin/cron=C2=A0<br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"font-size:12pt;background-color:transparent">-
 You might try enabling extra debugging information with the -x
 option.</span><br></p><p style=3D"line-height:100%;margin-bottom:0in;backgr=
 ound:transparent"><span style=3D"background-color:transparent">I tried one
 iteration with debug flags and captured logs but saw nothing useful
 there.</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">- </span><span style=3D"background=
 -color:transparent;font-variant-numeric:normal;font-variant-east-asian:norm=
 al"><font color=3D"#222222"><font face=3D"Arial, Helvetica, sans-serif"><fo=
 nt style=3D"font-size:12pt">What
 happens if you disable particular cron entries, like the script meant
 to run every minute?</font></font></font></span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">I will try lowering
 the frequency, after doing a reboot and seeing if/when the issue
 recurs. It seems this might be a =E2=80=9Cslow leak=E2=80=9D that will take
 patience to track.</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">I investigated
 further and found hangs on both top and vmstat, at varying times.</span><br=
 ></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">For vmstat, the
 first line (summary) is returned, but then nothing:</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">n0b:jim&gt; date ;
 vmstat 1 10</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Tue =
 May 17 13:09:14
 UTC 2022</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> pro=
 cs    memory   =20
  page                       disks   faults      cpu</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> r b=
       avm    fre
  flt  re  pi   po   fr   sr l0 n0   in   sy  cs us sy id</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> 1 0=
    304608  88784
   23   0   0    0    0    0  0  0 8882   44  14  0  1 99</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">^C</=
 p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">n0b:=
 jim&gt; date</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Tue =
 May 17 13:09:45
 UTC 2022</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">That stall is
 inconsistent though, as the results today are nominal:</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">n0b:jim&gt; date</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Fri =
 May 20 12:58:26
 UTC 2022</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">n0b:=
 jim&gt; vmstat 1
 3</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> pro=
 cs    memory   =20
  page                       disks   faults      cpu</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> r b=
       avm    fre
  flt  re  pi   po   fr   sr l0 n0   in   sy  cs us sy id</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> 1 0=
    310320  82568
   22   0   0    0    0    0  0  0 8870   43  13  0  1 99</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> 0 0=
    310320  82568
    0   0   0    0    0    0  0  0 8826   32  11  0  1 99</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"> 0 0=
    310320  82568
    0   0   0    0    0    0  0  0 8902   30  10  0  1 99</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">n0b:=
 jim&gt; date</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Fri =
 May 20 12:58:36
 UTC 2022</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">n0b:=
 jim&gt;=20
 </p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">The top command
 starts up, displays some data, but then does not refresh. The data
 are incomplete (values are all 0):</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">load averages:=20
 0.01,  0.02,  0.00;               up 11+21:37:48                    =20
        13:06:53</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">46 p=
 rocesses: 44
 sleeping, 2 on CPU</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU0=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU1=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU2=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU3=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Memo=
 ry: 298M Act,
 104K Inact, 12M Wired, 15M Exec, 259M File, 86M Free</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Swap=
 :=20
 </p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><br>
 
 </p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Like=
  vmstat, top
 worked later (except one core shows all zeroes).</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">load averages:=20
 0.01,  0.02,  0.00;               up 14+21:32:34       13:01:39</span><br><=
 /p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">50 p=
 rocesses: 48
 sleeping, 2 on CPU</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU0=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  1.6% interrupt, 98.4% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU1=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU2=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  0.0% interrupt,  0.0% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">CPU3=
  states:  0.0%
 user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Memo=
 ry: 303M Act,
 96K Inact, 12M Wired, 15M Exec, 262M File, 80M Free</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">Swap=
 :</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">However, cron
 commands have </span>not run<span style=3D"background-color:transparent"> s=
 ince.</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">My next steps will
 be:</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent"><spa=
 n style=3D"background-color:transparent">1. Reboot, taking
 note of initial state</span><br></p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">2. T=
 ry adding a swap
 device (have seen some odd Pi behavior with 0 swap)</p>
 <p style=3D"line-height:100%;margin-bottom:0in;background:transparent">3. D=
 ecrease the cron
 job frequency</p><p style=3D"line-height:100%;margin-bottom:0in;background:=
 transparent"><br></p><p style=3D"line-height:100%;margin-bottom:0in;backgro=
 und:transparent">Jim</p></div>
 
 --000000000000aaee7005df715c67--