Re: libelf

[David Holland <dholland-tech%netbsd.org@localhost>]

On Thu, Jan 14, 2010 at 08:46:50PM -0800, Jason Thorpe wrote:
 > > Any such strategic moves should be made with at least one eye towards
 > > moving on from ELF. It's not as set as all that and ELF has plenty of
 > > defects.
 > 
 > What would you suggest?  I don't see any practical alternative to
 > ELF out there now, nor on the horizon.  The only object format that
 > would even be close in terms of capabilities at this point would be
 > Mach-O, and that's impractical for a variety of reasons.

It'd have to be something new. (And isn't Mach-O a COFF derivative? Or
am I mixing things up? Anyway, it doesn't seem to particularly be a
step forward.)

It'd take some designing. I've done some of that over the years, but
it's been a while since the last round. So I'm not yet quite ready to
float a proposal, but I'm not that far from it either.

 > What are the defects in Elf as you see them?

Most of it is in shared library handling. The biggest issue is that
there's no way to specify transitive dependencies or coherence
requirements, so you can end up with broken processes.

E.g. consider libcurses, which re-exports libc's FILE * type. This
makes the libcurses ABI dependent on the libc ABI: a program using
curses can get a FILE * from libc and pass it to curses, which then
passes it back to libc. In this situation curses and the application
need to be using the same libc ABI, or it blows up. In particular, if
you make a change to the FILE * representation and create compat entry
points in libc, both curses and the application must use either the
new entry points or the old ones. This means you must recompile 
curses if you recompile the application, and not recompile curses
unless you recompile the application. Or you can bump the major
version of libcurses... but that only works for first-party libraries
where the bumps can be done synchronously. If you have the same ABI
dependence in a third-party library (grep finds a number of examples
in my /usr/pkg/include) you're out of luck.

The glibc people got badly burned by this problem years back when they
decided that because ELF 'symbol versioning' let them make compat
entry points they'd replace the entire stdio package. Bzzt...

Fixing this *properly* (so it just works) is a big undertaking; you
need to allow libcurses to be compiled against both libc ABIs at once,
and the resulting image needs to provide two sets of entry points that
propagate to the two sets of libc entry points. This also needs to
happen at compile/link time without much if any input from the library
maintainer, who for third party libraries can't realistically be
expected to keep up with every OS's random library compat issues.
This is perfectly feasible in principle, and maybe even possible with
llvm, but not likely to happen any time soon.

However, making an annotation that says that the libc used by the
libcurses client must be the same as the libc used by libcurses
itself... that's not very hard. That would at least prevent broken
binaries from running. (With ELF, they'll cheerfully run and exhibit
undefined behavior.)

A related problem is that if you bump the major version of libc and
don't recompile curses (but do recompile some application) is that you
can end up with load images that have multiple different libcs. This
might in principle stand a chance of working (except that most malloc
implementations will get confused by not having exclusive access to
sbrk) if ELF didn't bind symbols willy-nilly by name regardless of
what library they were supposed to be coming from. Which means that if
you have a process with two different copies of libc in it, only one
of them will actually be used, which *will* cause whatever wanted the
other copy to do the wrong thing.

Reckless binding by name causes other problems as well; see for
example the elf-lossage-demo tarballs in
http://www.hcs.harvard.edu/~dholland/computers/files/misc/.

This is not all, but it's a start.

Meanwhile, on the feature side instead of the bug side: the other big
thing we ought to have that ELF doesn't support is ResEdit. Why
shouldn't you be able to e.g. edit the compiled-in paths in an
executable, bind in images and change them later, or whatever? There's
no need to adopt Apple resource forks (or any of their imitations) to
get such functionality for executable files; executable files already
have all the necessary structures and indexes. All it takes is some
administrative reorganization and some strengthening of the way
sections work. And keeping full relocation data in output executables,
but that's no bad thing.

(ELF is a "don't get me started" topic. Can you tell? :-/ )

 > > btw, your mailer is sending very long lines.
 > 
 > Eh, it's just the Mail app on Snow Leopard ... Most MUAs I've seen
 > these days wrap the lines in a reasonable fashion, although some
 > mailing list software biffs the formatting.

Apparently so. There's no "format: flowed" in the headers here, or
whatever the precise notation is, and there should be.

-- 
David A. Holland
dholland%netbsd.org@localhost