Feb 2022
Redirecting stdout and stderr to files or to each other is something
we do everyday in our shell. But up until recently I never had to send
stderr to a pipe while keeping stdout on the terminal. And because
redirection syntax can be kinda weird sometimes, it wasn’t immediately
obvious to me how this would work.
My initial web searches didn’t immediately yield satisfactory answers, so I thought I’d write this short Bit to show the required syntax.
Let’s test things out using a script that writes to both stdout and
stderr:
# cat out.sh
>&2 echo "DEBUG: Starting up..."
>&2 echo "INFO: System check..."
echo "Real output"
Now if our debug output is too verbose but we can’t turn it off, we might
want to filter out all the lines on stderr that start with DEBUG:.
That’s the easy part (grep -vv ^DEBUG:), but how to we get stderr
into the pipe without affecting stdout?
In fish, the syntax is actually straightforward. Redirecting stderr
to a pipe uses the 2>| syntax. So with our example above it would be
something like:
$ ./out.sh 2>|grep -vv ^DEBUG:
Real output
INFO: System check...
The order in which the lines appear in the terminal differs from the script because the two lines take different routes until they reach our terminal, but that’s fine in this case.
We can obviously also redirect the result of that pipe to a file:
$ ./out.sh 2>|grep -vv ^DEBUG: > err.log
Real output
And we can still redirect the script’s stdout to a file as well:
$ ./out.sh > out.log 2>|grep -vv ^DEBUG: > err.log
Unfortunately, zsh doesn’t have, to my knowledge, any
convenient way to do this. There is no 2>| syntax or anything
equivalent. So we need to try something else.
In both zsh and bash there is a |& operator which is a shorthand
for 2>&1 |. So this is not quite what we want: It
redirects stderr to stdout and then both to the pipe. In our example
above this might not matter, but it doesn’t solve the general problem.
We can test how this fails our requirement by adding another pattern to our grep invocation:
$ ./out.sh |& grep -vv -e ^DEBUG: -e output
INFO: System check...
As we can see, the stdout was sent to the pipeline and filtered
through grep.
Instead of using a pipe to do the filtering of stderr we can also
spawn a subshell and connect its stdin to the stdout or stderr of
some other process.
$ ./out.sh 2> >(grep -vv ^DEBUG:)
Real output
INFO: System check...
This sends our script’s stderr to the subshell. We can verify that
stdout goes straight to the terminal by trying to filter it out like
above:
$ ./out.sh 2> >(grep -vv -e ^DEBUG: -e output)
Real output
INFO: System check...
stdout does not appear to go to the subshell. Great. The fact that
this uses a subshell instead of a pipe seems completely fine to me for
basically any case in which I need to do something like this.
The same theoretically works in bash, but not quite:
$ ./out.sh 2> >(grep -vv -e ^DEBUG:)
Real output
$ INFO: System check...
The redirection works as expected but (at least on my system, right now) bash is ready to show the prompt before the subshell has flushed its output and exited. That should not be a problem if the output of the subshell gets redirected to a file, but it is mildly annoying when the subshell is connected to the terminal.
The above might work fine when everything is written to a file anyway
but not so nice when working interactively in the terminal. What we can
do alternatively is use another file descriptor to basically exchange
stdout and stderr:
$ ./out.sh 3>&1 >&2 2>&3 |grep -vv -e ^DEBUG: -e output
Real output
INFO: System check...
That looks only moderately elegant and does not do exactly what we want,
but it basically works okay. We create a new file descriptor 3 that
points to stdout, we redirect the old stdout into stderr and the
old stderr to 3. So now, the script’s stderr ends up on the
file descriptor that will be connected to the pipe, and the script’s
stdout ends up on stderr. (We could of course also route the old
stdout to a file instead).
Out of the popular shells bash, zsh and fish only the latter seems
to have explicitly support for redirecting stderr into a pipe without
affecting stdout. For zsh and bash we can find workarounds that
might be fine depending on the specific use case.
I haven’t looked at how to do this in other shells, and I can’t
guarantee that I haven’t missed something about bash’s and zsh’s
redirection capabilities.
If there’s another way to make this work, I’d love to find out. So please let me know if you know of other redirection shenanigans that can be used to solve this!
See also: