Re: powerpc64 rescue/rescue link failures

[Matt Thomas <matt%3am-software.com@localhost>]

> On Jan 11, 2015, at 8:23 PM, Dennis Ferguson <dennis.c.ferguson%gmail.com@localhost> wrote:
> 
> 
> On 10 Jan, 2015, at 17:24 , Christos Zoulas <christos%astron.com@localhost> wrote:
>>> I can fix this by brute force with the attached patch, which
>>> inlines __cerror() everywhere it is used, but I was wondering
>>> if there was a better way.  I see .hidden sometimes being used
>>> for this but I have no idea if, or how, that might help this
>>> case.
>> 
>> The way this has been done so far is similar, basically giving each
>> library with syscall stubs its own copy of cerror...
>> 
>> find /usr/src/lib -name Makefile\* -exec grep cerror {} +
>> /usr/src/lib/libc/sys/Makefile.inc:     syscall.S __syscall.S __clone.S cerror.S
>> /usr/src/lib/libposix/sys/Makefile.inc:SRCS+=           cerror.S
>> /usr/src/lib/libposix/sys/Makefile.inc:CPPFLAGS+=       -D__cerror=__posix_cerror -I${LIBCDIR} -D_REENTRANT
>> /usr/src/lib/librt/sys/Makefile.inc:SRCS+=              cerror.S
>> /usr/src/lib/librt/sys/Makefile.inc:CPPFLAGS+=  -D__cerror=__rt_cerror -I${LIBCDIR} -D_REENTRANT
> 
> So after reading through
> 
>    http://refspecs.linuxfoundation.org/ELF/ppc64/PPC-elf64abi-1.9.html
> 
> I decided brute force is probably the only way to fix rescue/rescue,
> and is probably a good idea anyway.
> 
> I'll write down the reasoning in case I'm wrong.  For the branch from
> the syscall stub to __cerror() to work two things need to be true:
> 
> 1. __cerror() must be reachable from the syscall stub with a pc-relative
>   branch.
> 2. Both the stub and __cerror() need to be linked to use the same TOC.
> 
> If either of these is not true then ld needs to generate a stub for
> the call to use instead, and that stub must be called with a regular
> branch-and-link function call rather than a branch that doesn't return
> to the call site.
> 
> Making sure both the syscall stub and __cerror are in the same executable
> or shared object should almost always be sufficient to satisfy 1. (the span
> of a pc-relative branch is 64 MB), but is sufficient for 2. only if the
> shared object or executable has a single TOC.  Since the size of a TOC
> is limited to 64 kB an executable that needs more than that will have
> more than one TOC and will need to make stubbed calls internally as well.
> That's the problem with rescue:
> 
> b1$ powerpc64--netbsd-objdump -h lib/libc.so | grep .got 
> 20 .got          000086f8  000000000021a4c8  000000000021a4c8  0020a4c8  2**3
> b1$ powerpc64--netbsd-objdump -h usr/lib/libposix.so | grep .got
> 20 .got          00000068  0000000000011388  0000000000011388  00001388  2**3
> b1$ powerpc64--netbsd-objdump -h usr/lib/librt.so | grep .got    
> 21 .got          00000098  0000000000013aa0  0000000000013aa0  00003aa0  2**3
> b1$ powerpc64--netbsd-objdump -h rescue/cp | grep .got        
> 11 .got          0003fc38  00000000107ab730  00000000107ab730  006fb730  2**3
> 
> Rescue needs more than one TOC, making rescue work with the branch
> would seem to require some linker magic to make all syscall stubs
> in the executable end up with the same TOC as __cerror(), and that
> linker magic doesn't seem to exist.
> 
> So, assuming that's true, that leaves converting syscall stubs to
> branch-and-link to __cerror() instead or just inlining __cerror() in
> the syscall code as the only options for rescue (and maybe libc if
> it grows enough to have the problem too; it's over half way there
> already).  Doing a branch-and-link isn't sensible, however, since
> that also requires building a stack frame prior to the call which
> is the majority of the work the _REENTRANT branch of the __cerror()
> code is doing anyway.  Just including the __cerror() code in the
> syscall stub is only a few instructions more, and eliminates
> __cerror() altogether.
> 
> The nice side effect of doing this is that all other
> powerpc64-specific workarounds for this should now, in principle,
> be unnecessary.  I don't know if the cerror copies were also
> needed for some other architecture (like 32-bit powerpc?) though.

I have a fix for this in a tree on a box waiting for a new power supply.