libedit: patch to fix unexpected behavior on Linux

[Clint Byrum <clint%ubuntu.com@localhost>]

Hi there, I have been debugging this libedit issue for a little while
now, and I think I've got it.

With GNU readline,  using_history() doesn't actually take over the
terminal (it barely does anything actually). However, with libedit, it
does. This means that any program that runs using_history() as part of
its initialization, will take over the tty attached to stdin
immediately.

The issue with this is that sometimes you are running with stdout
directed to a pipe or file, and do not want/can't use editing. In fact,
it shouldn't be enabled.

This is made apparent when on Ubuntu Linux, you install the php package,
which has libedit linked in, and run


php | less

This leads to not being able to use the arrow keys in less, because php
grabs the controlling terminal. If PHP is built with readline, this
doesn't happen.

For whatever reason, NetBSD does not suffer from this problem. I
narrowed it down to a difference in the way tcsetattr(,TCSADRAIN) is
handled. in libedit/tty.c, tty_setup() calls this (via the tty_setty
macro), which on NetBSD, simply flushes and moves on. On Linux however,
linux takes this as a signal to grab the fd as a controlling terminal
and hold on to it.

So this problem probably isn't apparent on NetBSD, but it definitely
affects all Linux users of libedit.

I propose this patch be applied to libedit, which will disable tty
handling if the output is not a tty.

This is also tracked in Ubuntu here:

https://launchpad.net/bugs/322214

Your time and feedback on this matter is greatly appreciated!

=== modified file 'libedit/tty.c'
--- libedit/tty.c       2008-06-14 17:13:08 +0000
+++ libedit/tty.c       2011-01-26 21:03:59 +0000
@@ -472,6 +472,9 @@
        if (el->el_flags & EDIT_DISABLED)
                return (0);
 
+       if (!isatty(fileno(el->el_outfile)))
+               return (-1);
+
        if (tty_getty(el, &el->el_tty.t_ed) == -1) {
 #ifdef DEBUG_TTY
                (void) fprintf(el->el_errfile,