Re: NetBSD syzbot broken?

["'Aleksandr Nogikh' via syzkaller-netbsd-bugs" <syzkaller-netbsd-bugs%googlegroups.com@localhost>]

On Thu, Dec 26, 2024 at 4:24 PM Taylor R Campbell <riastradh%netbsd.org@localhost> wrote:
>
> > Date: Thu, 26 Dec 2024 10:26:29 +0100
> > From: Aleksandr Nogikh <nogikh%google.com@localhost>
> >
> > On Fri, Dec 20, 2024 at 2:01 AM Taylor R Campbell <riastradh%netbsd.org@localhost> wrote:
> > >
> > > Looks like the NetBSD syzbot is still not running, but for different
> > > reasons now:
> > >
> > > https://syzkaller.appspot.com/bug?extid=848e5b707cc911337a29 looks
> > > like a bug in the NetBSD build which was fixed back in October by
> > > adding shquote(3) to libnbcompat during the tools build
> >
> > Does something need to be done on the syzbot side to incorporate this fix?
>
> Shouldn't be -- if you just do a clean build of NetBSD with build.sh,
> starting with build.sh tools, it should pick that up.
>
> If you had cached the tools build, it's possible you need to delete
> your tooldir and rebuild it.  A quick skim of
> https://github.com/google/syzkaller/blob/d3ccff6372e07c6aabd02b5da419aa6492b5f0ad/pkg/build/netbsd.go#L24
> suggests you do use fresh tools:
>
>         // Clear the tools.
>         if _, err := osutil.RunCmd(5*time.Minute, params.KernelDir, "rm", "-rf", "obj/"); err != nil {
>                 return ImageDetails{}, err
>         }
>
> So this should be addressed, but if it still happens (e.g., if I
> misunderstood what is getting cleaned here -- usually I use `build.sh
> -O ../obj' so all the build products go into ../obj and the source
> tree can even be read-only), I can take a closer look.

Looked once more at https://syzkaller.appspot.com/bug?extid=848e5b707cc911337a29

I think it was also a transient problem -- there was a spike on Oct
31st and on Dec 20th, and that's all. Since we have a "test error" for
every recent NetBSD commit, syzbot can build NetBSD just fine. So I
guess nothing needs to be done here.

>
> > > https://syzkaller.appspot.com/bug?extid=85f03a4da9f6aedf4c44 looks
> > > like a golang build problem that probably isn't NetBSD specific?
> > >
> > > --- FAIL: TestFuzz (11.34s)
> > >     testutil.go:35: seed=1733770450604578864
> > >     fuzzer_test.go:210: CRASH: first bug
> > >     fuzzer_test.go:210: CRASH: second bug
> > >     fuzzer_test.go:88: resulting corpus:
> > >     testing.go:1231: TempDir RemoveAll cleanup: unlinkat /tmp/TestFuzz1015377258/001: directory not empty
> > > FAIL
> >
> > This looks like a one-off reincarnation of
> > https://github.com/google/syzkaller/issues/4920
> > It may cause syzbot sometimes to not update to a newer revision, but
> > it should not prevent it from doing fuzzing.
>
> OK, thanks!
>
> > > https://syzkaller.appspot.com/bug?extid=ed1ef9cf91c3de65d44b looks
> > > like a testbed problem with posix_spawnp -- presumably this is
> > > happening before anything even starts running NetBSD?
> >
> > Yes, it happens when syzbot tries to update to a newer NetBSD kernel
> > revision. It runs syz-manager with a "-mode=smoke-test" option and
> > that quickly fails with "SYZFAIL: posix_spawnp failed".
> >
> > This must have begun after
> > https://github.com/google/syzkaller/commit/bc1a1b50f942408a9139887b914f745d9fa02adc
> >
> > > Can I trouble any of you folks to help get syzbot again on NetBSD?
> > > Anything I can do to help make it happen?
> >
> > It would really help if you could take a look at the posix_spawnp()
> > invocation in the commit I referenced above. On NetBSD, the syscall
> > fails with "errno 22: Invalid argument". What could be that invalid
> > argument? Why was posix_spawn() working fine?
>
> That's pretty weird, because posix_spawnp just searches $PATH and then
> defers to posix_spawn:

I'm sorry, it looks like I misinformed you about the
https://github.com/google/syzkaller/commit/bc1a1b50f942408a9139887b914f745d9fa02adc
commit.

The commit has indeed caused "netbsd test error: SYZFAIL: posix_spawnp
failed" to happen, but that actually only replaced an older problem
that already existed. It just had a different title: "netbsd test
error: SYZFAIL: posix_spawn failed", see
https://syzkaller.appspot.com/bug?extid=6b8de84f7954fa8b4ecb

And that one appeared ~ on Jun 25th:
https://syzkaller.appspot.com/netbsd/graph/crashes?regexp=netbsd+test+error%3A+SYZFAIL%3A+posix&Months=10&show-graph=Show+graph

That's actually around the time when the corresponding code snippet
first appeared in syzkaller.

Before we start deep debugging of this, does the code ring any bells
for you now given that the problem is not in the posix_spawn ->
posix_spawnp change? :)

>
> https://nxr.netbsd.org/xref/src/lib/libc/gen/posix_spawnp.c?r=1.5
>
> Only the file -> fpath argument is changed (and then only if there's
> no slash in the pathname); all of the other ones are passed through
> verbatim to posix_spawn.
>
> If you can ktrace or ktruss the program it might help to find what
> arguments are being passed to posix_spawn.  The usage is:
>
> ktrace ./prog        # writes binary trace to ktrace.out
> kdump                # formats binary trace in ktrace.out human-readably
>
> or, if you want a usage model a little more like strace at somewhat
> higher overhead:
>
> ktruss [-o <outfile>] ./prog        # traces and formats simultaneously
>

I fear it might be not so straightforward to get meaningful data out
of syz-executor traces. It does so much initialization and forking +
requires following a complicated communication protocol with the host
process even to get to the stage where it would call posix_spawnp().

If we still want to, we can try to do it by
1) Hacking https://github.com/google/syzkaller/blob/f3558dbf032eab2b77c1cb11b9ce2baffe7838d3/pkg/rpcserver/local.go#L67
to prepend ktrace/ktruss to the syz-executor call.
2) Building syz-execprog on the hacked syzkaller and using it to run
some simple program inside a VM.

**But** if we can get the needed information just by adding extra
`debug(...)` logging calls into syz-executor code, that would be
incomparably easier than everything else.

> (I guess I should try going through the instructions to run syzkaller
> locally at this point rather than just teledebug everything remotely!)

-- 
You received this message because you are subscribed to the Google Groups "syzkaller-netbsd-bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-netbsd-bugs+unsubscribe%googlegroups.com@localhost.
To view this discussion visit https://groups.google.com/d/msgid/syzkaller-netbsd-bugs/CANp29Y493jb-7%3DJDKSihf4q%2BNHZ2Y5-An9J1umh4riVifzRRJw%40mail.gmail.com.