lib/58955: libedit: rl_event_hook not called when output is piped to another process

[mrrosen%alumni.cmu.edu@localhost]

>Number:         58955
>Category:       lib
>Synopsis:       libedit: rl_event_hook not called when output is piped to another process
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    lib-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Fri Jan 03 00:00:00 +0000 2025
>Originator:     Michael Rosen
>Release:        
>Organization:
>Environment:
Linux --- 5.15.153.1-microsoft-standard-WSL2+ #11 SMP Mon Apr 29 14:24:57 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux
>Description:
rl_event_hook should be called periodically if set while waiting on characters during blocking readline calls. From the documentation on GNU readline:

"If non-zero, this is the address of a function to call periodically when Readline is waiting for terminal input. By default, this will be called at most ten times a second if there is no keyboard input."

However, if the output of a program using Libedit's readline implementation is piped into another program (ie ./libedit_readline_program_using_rl_event_hook | grep "some output"), the program will not call the rl_event_hook function (which might include some important logic) and the readline call will just block, possibly forever.

The problem arises from the following condition to the rl_event_hook installation in readline (readline.c:467):

if (rl_event_hook && !(e->el_flags & NO_TTY)) {
    el_set(e, EL_GETCFN, _rl_event_read_char);
    used_event_hook = 1;
}

The check for NO_TTY (which if one dives into where thats set comes from tty.c:508, isatty(el->el_outfd), which is false when the output is piped to another process) is the reason the rl_event_hook will not be installed in this case.

Is there a reason the output needs to be a tty? Removing this condition seems to fix the problem but I would assume there was a reason for it to be there in the first place (no comment nor any information in the commit message for the change that introduced it, revision 1.38 of readline.c).

>How-To-Repeat:
Using this program:

#include <stdio.h>
#include <stdlib.h>

#include <editline/readline.h>

int try_rl_event_hook(void) {
    static short hit = 0;

    if (hit++ == 0) {
        fprintf(stderr, "in try_rl_event_hook\n");
    }

    return 0;
}

int main() {
    rl_event_hook = try_rl_event_hook;

    while (1) {
        char *input = readline("prompt> ");
        if (!input) {
            break;
        }
        printf("got input: %s\n", input);
        free(input);
    }

    return 0;
}

If run by itself will print "in try_rl_event_hook" frequently, if run piped into grep or some other process/file will not.
>Fix:
Remove the condition checking for tty:


if (rl_event_hook) {
    el_set(e, EL_GETCFN, _rl_event_read_char);
    used_event_hook = 1;
}