port-amd64/53316: syscalls with more than 6 args may not work

[martin%NetBSD.org@localhost]

>Number:         53316
>Category:       port-amd64
>Synopsis:       syscalls with more than 6 args may not work
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    port-amd64-maintainer
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat May 26 12:00:00 +0000 2018
>Originator:     Martin Husemann
>Release:        NetBSD 8.99.18
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD night-owl.duskware.de 8.99.18 NetBSD 8.99.18 (NIGHT-OWL) #600: Sat May 26 09:22:08 CEST 2018 martin%night-owl.duskware.de@localhost:/usr/src/sys/arch/amd64/compile/NIGHT-OWL amd64
Architecture: x86_64
Machine: amd64
>Description:

The new test /usr/tests/lib/libc/sys/t_syscall fails on amd64. It tries
to call mmap(2) via __syscall(2) with this code:

        p = (const char *)__SYSCALL_TO_UINTPTR_T(__syscall(SYS_mmap,
                0, sizeof(secrect_data), PROT_READ, MAP_PRIVATE, fd, 0, 0, 0));

This ends up in the kernel in src/sys/arch/x86/x86/syscall.c here:

    119 #ifdef __x86_64__
    120         /*
    121          * The first 6 syscall args are passed in rdi, rsi, rdx, r10, r8 and r9
    122          * (rcx gets copied to r10 in the libc stub because the syscall
    123          * instruction overwrites %cx) and are together in the trap frame
    124          * with space following for 4 more entries.
    125          */
    126         if (__predict_false(callp->sy_argsize > 6 * 8)) {
    127                 error = copyin((register_t *)frame->tf_rsp + 1,
    128                     &frame->tf_arg6, callp->sy_argsize - 6 * 8);
    129                 if (error != 0)
    130                         goto bad;
    131         }

In this case, callp->sy_argsize is 72 (bytes), so 9 arguments, 6 of which
come in registers and already are in the trapframe (and thos work fine).
The 3 additional args are copied in from one longword below the current
stack, and verifying with gdb in userland shows the copyin() getting the
right values from the stack.

The syscall stub in libc is plain RSYSCALL. I don't know the amd64 off
hand good enough, maybe this is a variadic function only issue.

The data copied in and the 6 working args match what ktrace prints
about the call.

The test program works fine on various other architectures, including some
64bit ones.

>How-To-Repeat:

cd /tmp
ktrace /usr/tests/lib/libc/sys/t_syscall mmap___syscall
kdump

and watch for this mmap right after a 1k write to the test file:

        \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
        \0\0\0"
  2071      1 t_syscall RET   write 1024/0x400
  2071      1 t_syscall CALL  mmap(0,0x400,PROT_READ,0x2<PRIVATE,,,>,3,0,0x7f7f00000000)
  2071      1 t_syscall RET   mmap 140187597225984/0x7f7ff7ef4000

Note that the last arg is wrong, so the eroneous offset requested in the mmap
leads to a sigbus as we are accessing the resulting map beyound EOF.

>Fix:
n/a