Re: powerpc64 rescue/rescue link failures

[Dennis Ferguson <dennis.c.ferguson%gmail.com@localhost>]

On 10 Jan, 2015, at 17:24 , Christos Zoulas <christos%astron.com@localhost> wrote:
>> I can fix this by brute force with the attached patch, which
>> inlines __cerror() everywhere it is used, but I was wondering
>> if there was a better way.  I see .hidden sometimes being used
>> for this but I have no idea if, or how, that might help this
>> case.
> 
> The way this has been done so far is similar, basically giving each
> library with syscall stubs its own copy of cerror...
> 
> find /usr/src/lib -name Makefile\* -exec grep cerror {} +
> /usr/src/lib/libc/sys/Makefile.inc:     syscall.S __syscall.S __clone.S cerror.S
> /usr/src/lib/libposix/sys/Makefile.inc:SRCS+=           cerror.S
> /usr/src/lib/libposix/sys/Makefile.inc:CPPFLAGS+=       -D__cerror=__posix_cerror -I${LIBCDIR} -D_REENTRANT
> /usr/src/lib/librt/sys/Makefile.inc:SRCS+=              cerror.S
> /usr/src/lib/librt/sys/Makefile.inc:CPPFLAGS+=  -D__cerror=__rt_cerror -I${LIBCDIR} -D_REENTRANT

So after reading through

    http://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi-1.9.html

I decided brute force is probably the only way to fix rescue/rescue,
and is probably a good idea anyway.

I'll write down the reasoning in case I'm wrong.  For the branch from
the syscall stub to __cerror() to work two things need to be true:

1. __cerror() must be reachable from the syscall stub with a pc-relative
   branch.
2. Both the stub and __cerror() need to be linked to use the same TOC.

If either of these is not true then ld needs to generate a stub for
the call to use instead, and that stub must be called with a regular
branch-and-link function call rather than a branch that doesn't return
to the call site.

Making sure both the syscall stub and __cerror are in the same executable
or shared object should almost always be sufficient to satisfy 1. (the span
of a pc-relative branch is 64 MB), but is sufficient for 2. only if the
shared object or executable has a single TOC.  Since the size of a TOC
is limited to 64 kB an executable that needs more than that will have
more than one TOC and will need to make stubbed calls internally as well.
That's the problem with rescue:

b1$ powerpc64--netbsd-objdump -h lib/libc.so | grep .got 
 20 .got          000086f8  000000000021a4c8  000000000021a4c8  0020a4c8  2**3
b1$ powerpc64--netbsd-objdump -h usr/lib/libposix.so | grep .got
 20 .got          00000068  0000000000011388  0000000000011388  00001388  2**3
b1$ powerpc64--netbsd-objdump -h usr/lib/librt.so | grep .got    
 21 .got          00000098  0000000000013aa0  0000000000013aa0  00003aa0  2**3
b1$ powerpc64--netbsd-objdump -h rescue/cp | grep .got        
 11 .got          0003fc38  00000000107ab730  00000000107ab730  006fb730  2**3

Rescue needs more than one TOC, making rescue work with the branch
would seem to require some linker magic to make all syscall stubs
in the executable end up with the same TOC as __cerror(), and that
linker magic doesn't seem to exist.

So, assuming that's true, that leaves converting syscall stubs to
branch-and-link to __cerror() instead or just inlining __cerror() in
the syscall code as the only options for rescue (and maybe libc if
it grows enough to have the problem too; it's over half way there
already).  Doing a branch-and-link isn't sensible, however, since
that also requires building a stack frame prior to the call which
is the majority of the work the _REENTRANT branch of the __cerror()
code is doing anyway.  Just including the __cerror() code in the
syscall stub is only a few instructions more, and eliminates
__cerror() altogether.

The nice side effect of doing this is that all other
powerpc64-specific workarounds for this should now, in principle,
be unnecessary.  I don't know if the cerror copies were also
needed for some other architecture (like 32-bit powerpc?) though.

Dennis Ferguson