the zen of SIGPIPE

["James K. Lowden" <jklowden%schemamania.org@localhost>]

(Please see http://gnats.netbsd.org/46021 for background information.
The PR was closed because the problem was not clearly enough stated and
agreement on a solution did not exist.  This message hopes to correct
that situation.)  

We have a small number of executables in /usr/bin that write an error
message to standard error when they receive SIGPIPE on writing to
standard output.  These executables are by far in the minority and
their behavior is quite recent.  

I contend no command-line utility should write such a message, and that
any that do share a common bug.  That's an aesthetic judgement,
insofar as the issue is not how the utility terminates, but what
messages it emits.  

First, let us consider normal behavior.  Examples include ls(1) and 
cat(1).  

$ ls /etc/*.conf | head -1
/etc/daily.conf

$ cat /etc/*.conf | head -1
#       $NetBSD: daily.conf,v 1.4 2000/10/01 05:53:01 lukem Exp $

Each wrote to standard output, which was read by head(1).
Each was terminated early by SIGPIPE, and went away quietly.  /bin/sh
considers the pipeline successful, setting $? to 0, because the last
element in the pipeline returned a zero status.  

Now consider the inexcusable example of xargs(1).  What happens if xargs
is introduced into the pipeline?

$ echo /etc/*.conf | xargs cat | head -1
#       $NetBSD: daily.conf,v 1.4 2000/10/01 05:53:01 lukem Exp $
xargs: cat terminated by SIGPIPE

Hmm.  cat was terminated by SIGPIPE; that fact is carried back to the
invoking process by the kernel.  If the invoking process is /bin/sh,
it's handled normally, i.e. silently.  If the invoker is xargs(1), it
reports the exit status on standard error.  

I think we can agree that's inconsistent.  Either the exit value of the
intermediate pipeline element is important, or it is not.  Either it
needs to be reported, or it does not.  

Yes, I understand that /usr/bin/xargs is not /bin/sh, and that they're
allowed to have different behavior.  That's a magnificently small
accident, though: What is and isn't "the shell" is part cumulative
judgement and part happenstance.  

xargs is an integral part of the shell.  It happens not be built-in,
but it exists to augment the shell and has no other use.  Nothing
prevents making it a built-in.  Some shells require e.g. /bin/test;
others have it built in for efficiency.  (BTW the sh manual doesn't
indicate "test" is a built-in, but "type test" does.)  The boundary
between what's "the shell" and what's not is thus both thin and
arbitrary. What's more important than the number of binaries or
processes is their consistent, cohesive behavior.  

Now let us consider *why* /bin/sh doesn't report SIGPIPE for
intermediate pipeline processes, and why in fact most utilities
in /usr/bin don't report it.   

1.  It's common for a reader, e.g. head(1), not to read everything.  

2.  There is no mechanism, other than closing the pipe, for the reader
to notify the writer that it's done.  

Ergo, closing the pipe is *normal*, not an error.  

That's been recognized implicitly for decades.  The unix tradition of
ignoring SIGPIPE on standard output extends almost universally from 
arp to zcat. In fact, of all the other utilities in /usr/bin, only one
that I know of reports SIGPIPE errors when writing to standard output:
pax(1).  

$ pax -zf freetds-dev.0.92.486.tar.gz | head -1 
freetds-dev.0.92.486/m4/ac_caolan_func_which_gethostbyname_r.m4
pax: Listing incomplete. (Broken pipe)
gzip: error writing to output: Broken pipe

How unusual is that?  Let's count.  

$ echo 200 / $(ls /usr/bin | wc -l) | bc -l
.45871559633027522935

Of 436 files in /usr/bin, I've identified 0.5% as having that 
behavior; 99.5% do not.  That's either 434 bugs, or 2.  

The case against pax should not be controversial, but it is
somewhat subtle to fix because for pax SIGPIPE is *sometimes*
significant.  

pax surely needs to report SIGPIPE errors when it's sending an archive
over a pipe, for instance. But the above common use case -- listing the
files in an archive -- cannot lead to archive corruption or to some
files not being extracted.  pax is simply writing to standard output
and, like every other NetBSD utility, should keep mum if cut off
early.  

FWIW, the report-all-errors mantra has infected other systems and
utilities not in the NetBSD tree.  You may have encountered them in
pkgsrc.  My leading unfavorite is Subversion:

$ svn status -v External/ | head -1
               438      435 foomanchu    External
svn: Write error: Broken pipe

If only that were the greatest of Subversion's faults.  

The lesson is clear.  Command-line utilities should not emit an error
message if they receive SIGPIPE upon writing to standard output.  That
signal is a normal part of daily existence, as evinced by the fact that
the vast majority of utilities ignore it.  Reported as an error it only
adds clutter to the screen, and requires careful analysis to understand
its origin. 

xargs and pax should be changed to eliminate SIGPIPE notification.  Let
us agree to that, and to prevent changes to /usr/bin that add 
such messages the system, for the user's sake.  

--jkl