bin/54068: ATF tests hung in pipe2_consume

[gson%gson.org@localhost (Andreas Gustafsson)]

>Number:         54068
>Category:       bin
>Synopsis:       ATF tests hung in pipe2_consume
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Mar 21 11:15:00 +0000 2019
>Originator:     Andreas Gustafsson
>Release:        NetBSD-current, source date 2019.03.10.15.45.26, with patch
>Organization:

>Environment:
System: NetBSD guava.gson.org 7.2 NetBSD 7.2 (GENERIC.201808291900Z) amd64
Architecture: x86_64
Machine: amd64
>Description:

As I was running the ATF tests on real amd64 hardware, they hung after
printing "pipe2_consume: " in the ticker output.

That is, the machine is still up, but the ATF tests are no longer
making proress.  If a test case hangs, I would expect ATF to time out
the test in case and continue, typically after five minutes, but in
this case there has been no progress at all after half an hour or so.

ps shows:

UID   PID  PPID CPU PRI NI   VSZ    RSS WCHAN   STAT TTY      TIME COMMAND
  0   509   634   0  85  0 24500   3104 pipe_rd I+   tty00 0:00.33 atf-report -o ticker:- -o xml:/tmp/tests/test.xml 
  0   665   404 231  85  0 24528   3864 pipe_rd I+   tty00 0:04.35 atf-run 
  0 21753   665   0   0  0     0      0 -       Z    tty00 0:00.00 (t_pipe2)

gdb shows:

anita-test# gdb atf-run
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64--netbsd".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from atf-run...(no debugging symbols found)...done.
(gdb) attach 665
Attaching to program: /usr/bin/atf-run, process 665
Couldn't get registers: Device busy.
(gdb) [New LWP 1 of process 665]
Reading symbols from /usr/lib/libstdc++.so.8...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libm.so.0...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libc.so.12...(no debugging symbols found)...done.
Reading symbols from /usr/libexec/ld.elf_so...(no debugging symbols found)...done.
[Switching to LWP 1 of process 665]
0x00007936ee04287a in read () from /usr/lib/libc.so.12

(gdb) where
#0  0x00007936ee04287a in read () from /usr/lib/libc.so.12
#1  0x0000000048e16d0f in tools::io::muxer::read_one(unsigned long, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) ()
#2  0x0000000048e17259 in tools::io::muxer::flush() ()
#3  0x0000000048e12b22 in tools::test_program::run_test_case(tools::fs::path const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std
 ::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, tools::fs::path const&, tools::fs::path const&, tools::test_program::atf_tps_writer&) ()
#4  0x0000000048e0948c in atf_run::run_test_program(tools::fs::path const&, tools::test_program::atf_tps_writer&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) ()
#5  0x0000000048e0acf6 in atf_run::run_test(tools::fs::path const&, tools::test_program::atf_tps_writer&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) ()
#6  0x0000000048e0aedb in atf_run::run_test_directory(tools::fs::path const&, tools::test_program::atf_tps_writer&) ()
#7  0x0000000048e0ad2f in atf_run::run_test(tools::fs::path const&, tools::test_program::atf_tps_writer&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) ()
#8  0x0000000048e0aedb in atf_run::run_test_directory(tools::fs::path const&, tools::test_program::atf_tps_writer&) ()
#9  0x0000000048e0ad2f in atf_run::run_test(tools::fs::path const&, tools::test_program::atf_tps_writer&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) ()
#10 0x0000000048e0aedb in atf_run::run_test_directory(tools::fs::path const&, tools::test_program::atf_tps_writer&) ()
#11 0x0000000048e0ad2f in atf_run::run_test(tools::fs::path const&, tools::test_program::atf_tps_writer&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&) ()
#12 0x0000000048e28bab in atf_run::main() ()
#13 0x0000000048e1e4f9 in tools::application::app::run(int, char* const*) ()
#14 0x0000000048e28808 in main ()
(gdb) 

This is with source date 2019.03.10.15.45.26 + kre's patches to fix
the pmap related panic.  My logs show that the tests have also hung in
exactly the same place once before, with unpatched sources from
2019.02.12.10.16.58:

  http://www.gson.org/netbsd/bugs/build/amd64-baremetal/2019/2019.02.12.10.16.58/test.log

>How-To-Repeat:

>Fix: