kern/54727: writing a large file causes unreasonable system behaviour

[spz%NetBSD.org@localhost]

>Number:         54727
>Category:       kern
>Synopsis:       writing a large file causes unreasonable system behaviour
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Dec 01 07:00:00 +0000 2019
>Originator:     S.P.Zeidler
>Release:        NetBSD 9.0_RC1
>Organization:
	The NetBSD Foundation
>Environment:
System: NetBSD franklin.NetBSD.org 9.0_RC1 NetBSD 9.0_RC1 (FRANKLIN) #1: Sat Nov 30 15:09:30 UTC 2019 spz%franklin.NetBSD.org@localhost:/home/netbsd/9/amd64/obj/sys/arch/amd64/compile/FRANKLIN amd64
Architecture: x86_64
Machine: amd64
>Description:
	running "ssh othersystem dobackup > /storage/file" generating
        a large file while otherwise idle, causes the system to hang
        (no response in 5 minutes on serial console, ddb ps shows
        daemons memory starved or missing) or with 
        "sysctl -w kern.maxvnodes=10000", pgdaemon spinning several minutes
        before any new program can start.
        "vmstat 1" shows:
 procs    memory      page                       disks   faults      cpu
 r b      avm    fre  flt  re  pi   po   fr   sr s0 s1   in   sy  cs us sy id
 3 0 47032084  21836    2   7   0    0    0 28464  0  0   85   20  40  0  7 93
 0 3 47032084  21836    0   0   0    0    0    0  0  0  238   18  36  0  7 93
 1 2 47032168  21824    7  31   0    0    0 28966  0  0   84   76  48  0 14 86
 1 1 47032244  21836   37 240   0    0    0 28805  1  0   83  214  49  0  7 93
 0 2 47032244  21836    0   0   0    0    0    0  0  0   84  108  44  0  6 94
 0 2 47032056  21908   22 140   0    0    9 28954  7  0  137   44  64  0 12 88
 0 2 47032236  21896  101 117   0    0   35 28805  1  0  139  118  59  0  7 93
 0 2 47032236  21896    0   0   0    0    0    0  0  0   90   75  41  0  6 94
 0 2 47032300  21840   34 142   0    0    4 28495  2  0  123   60  57  0  7 93
 0 3 47032312  21872    5  65   0    0    0 29033  0  0   91  101  85  0 10 90
 0 3 47032312  21872    0   0   0    0    0    0  0  0   84   18  33  0  6 94
 0 2 47032312  21852  137 127   0    0    4 28621  0  1  139  323  55  0 15 85
 0 2 47032312  21852    0   0   0    0    0    0  0  0   83   75  37  0  6 94
 0 2 47032320  21764   11  22   0    0    0 28478  0  0   83   19  45  0 16 84
 0 2 47032320  21764    0   0   0    0    0    0  0  0   85   96  44  0  6 94
 0 2 47032380  21820   74 113   0    0    0 28837  0  0  119   87  50  0 11 89
 0 2 47032380  21820    0   0   0    0    0    0  0  0   86   18 132  0  6 94
 1 1 47032452  21828    9 127   0    0    0 28480  0  0   85  106  44  0 12 88
 0 3 47032452  21812    0   0   0    0    0    0  0  0   89   92  44  0 19 81
 0 3 47032472  21828    2  26   0    0    0 29009  1  0   88   25  42  0 13 87
 0 3 47030372  21828    0   0   0    0    0    0  0  1   91   18

top has:
load averages:  0.92,  1.73,  1.89;               up 0+15:20:54        06:38:57
291 threads: 65 idle, 210 sleeping, 16 on CPU
CPU states:  0.0% user,  0.0% nice, 11.2% system,  0.1% interrupt, 88.4% idle
Memory: 33G Act, 11G Inact, 16M Wired, 27M Exec, 45G File, 84M Free
Swap: 48G Total, 48G Free

  PID   LID USERNAME PRI STATE      TIME   WCPU    CPU NAME      COMMAND
    0   216 root     126 pgdaem/6 292:24 84.62% 84.62% pgdaemon  [system]
 1335     1 backups   84 select/6 284:58 12.55% 12.55% -         ssh
    0   217 root     124 syncer/1  59:05 11.77% 11.77% ioflush   [system]
  347     1 root      85 select/1  21:54  8.11%  8.11% -         blacklistd
    0   218 root     125 aiodon/3  13:01  8.06%  8.06% aiodoned  [system]
 3473     1 root      43 CPU/4      3:59  0.00%  0.00% -         top
 2761     1 root      85 select/5   3:41  0.00%  0.00% -         top
  363     1 xymon     85 nanosl/1   2:12  0.00%  0.00% -         xymonlaunch

when no program starts have been requested for a while, the system
calms down to:
load averages:  0.51,  0.66,  1.14;               up 0+15:32:16        06:50:19
287 threads: 66 idle, 205 sleeping, 16 on CPU
CPU states:  0.4% user,  0.0% nice,  2.2% system,  1.0% interrupt, 96.2% idle
Memory: 38G Act, 6619M Inact, 16M Wired, 27M Exec, 45G File, 85M Free
Swap: 48G Total, 48G Free

  PID   LID USERNAME PRI STATE      TIME   WCPU    CPU NAME      COMMAND
 1335     1 backups  116 tstile/9 288:23 27.20% 27.20% -         ssh
    0   216 root     126 CPU/6    295:01 20.07% 20.07% pgdaemon  [system]
    0   217 root     124 syncer/1  59:53  3.22%  3.22% ioflush   [system]
  347     1 root      85 select/8  21:58  0.00%  0.00% -         blacklistd
    0   218 root     125 aiodon/0  13:06  0.00%  0.00% aiodoned  [system]
 3473     1 root      43 CPU/13     4:00  0.00%  0.00% -         top
  363     1 xymon     85 nanosl/9   2:12  0.00%  0.00% -         xymonlaunch

>How-To-Repeat:
	dd if=/dev/zero of=file
>Fix:

>Unformatted:
 	20191130