Every process is initialized with three open file descriptors, stdin, stdout, and stderr. stdin is an abstraction for accepting input (from the keyboard or from pipes) and stdout is an abstraction for giving output (to a file, to a pipe, to a console).
That's a very simplified explanation but true nonetheless. Those three file descriptors are collectively called 'The Standard Streams'.
Where does stderr come from?
It's fairly straightforward to understand why stdin and stdout exist, however stderr seems like the odd one out. Why do we need another stream for errors?
This is a quote from Doug McIllroy, inventor of Unix pipes, explaining how stderr came to be. 'v6' is referring to a version of specific version of the original Unix operating system that was released in 1975.
All programs placed diagnostics on the standard output. This had always caused trouble when the output was redirected into a ï¬le, but became intolerable when the output was sent to an unsuspecting process. Nevertheless, unwilling to violate the simplicity of the standard-input-standard-output model, people tolerated this state of affairs through v6. Shortly thereafter Dennis Ritchie cut the Gordian knot by introducing the standard error ï¬le. That was not quite enough. With pipelines diagnostics could come from any of several programs running simultaneously. Diagnostics needed to identify themselves.Doug McIllroy, "A Research UNIX Reader: Annotated Excerpts from the Programmer’s Manual, 1971-1986"
Why do we need stderr?
So there was a time when stderr didn't exist. As
McIllroy mentions, in that time people were unwilling to violate the
simplicity of the standard-input-standard-output model to add something
like stderr. Let's have a look at the grep command to see a real example of why stderr is needed.
$ grep hosts /private/etc/*
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net -hosts -nobrowse,hidefromfinder,nosuid
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
From this command I've got some diagnostic messages (permission errors) mixed with real matched output. When writing to a console both stdout and stderr will be printed,
this is useful so that I can see the errors as they happen. But what if
I was redirecting the output to a file to send to somebody else, or
save for later?
$ grep hosts /private/etc/* > results.txt
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
$ cat results.txt
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net -hosts -nobrowse,hidefromfinder,nosuid
That's actually just what I wanted: I saw the errors printed to the
console but all of the matched output was sent to the file. In the time
of Unix v6 both the errors AND the output would have been sent to the
file, which I would have needed to clean up manually. Woohoo for stderr.
Redirecting stderr
You can also achieve the opposite, send stderr to a file and print stdout on the console by redirecting a specific file descriptor number. For stderr that's 2.
$ grep hosts /private/etc/* 2> error.log
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net -hosts -nobrowse,hidefromfinder,nosuid
$ cat error.log
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
And one more example, redirecting both stderr and stdout to a file, to preserve exactly what you'd see on the console.
$ grep hosts /private/etc/* &> results.txt
$ cat results.txt
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
/private/etc/amavisd.conf:# from internal hosts to a dedicated TCP port (such as 10026) for filtering
/private/etc/auto_master:/net -hosts -nobrowse,hidefromfinder,nosuid
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
Identify Yourself!
Notice how the error messages from grep are prepended with 'grep:'?
grep: /private/etc/master.passwd: Permission denied
This is what McIllroy was talking about when he said "With pipelines diagnostics could come from any of several programs running simultaneously. Diagnostics needed to identify themselves." Consider a pipeline like this:
grep hosts /private/etc/* | awk '{print($1)}' | xargs cat
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
cat: /private/etc/amavisd.conf:#: No such file or directory
cat: /private/etc/auto_master:/net: No such file or directory
Were it not for the prefixed command name in front of the error we'd
have no idea which command was printing the error. For all we know every
error could have been coming from awk. Thankfully most commands support this convention today so we can figure out where we're going wrong.
stderr and Pipes
When you pipe the output of one command to another command only the stdout
is passed over the pipe. Again, this is just what we want, but was not
the case back in the Unix v6 days. Have a look at this grep example:
$ grep hosts /private/etc/* | grep 'Permission denied' > results.txt
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
$ cat results.txt
So the errors were printed to the console but they're not present in
the output. Highlighted in this case by the fact that we searched for
them with the second grep command but they don't exist in the results file.
If you need to pass stderr over a pipe along with stdout you can do that by redirecting the stderr stream (file descriptor #2) to stdout (file descriptor #1), like so:
$ grep hosts /private/etc/* 2>&1 | grep 'Permission denied' > results.txt
$ cat results.txt
grep: /private/etc/AFP.conf: Permission denied
grep: /private/etc/aliases.db: Permission denied
grep: /private/etc/kcpassword: Permission denied
grep: /private/etc/krb5.keytab: Permission denied
grep: /private/etc/master.passwd: Permission denied
In this case there was nothing printed to the console, but the errors were matched and sent to the results file.
I probably don't want your help docs
It's pretty obvious that any error messages should be printed to stderr
because I probably don't want them as input to other programs, and if I
really do then I can use redirection to get it. But McIllroy
specifically says that stderr is for diagnostic messages, why didn't he just say errors?
stderr is also the place where you print stuff like help docs when an invalid argument is passed to a command. In this case its help docs should be printed, but not be passed on to the next program.
$ cat Gemfile | grep -k gem | awk '{print($2)}' | xargs gem install
grep: invalid option -- k
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
Thankfully the help docs for grep are printed to stderr so I don't try to install them as gems in this case.
But if I'm invoking a command specifically to read its help docs, then it should be printed to stdout. Take the following example:
$ git --help | grep pull
In this case I certainly want the help docs; I asked for them explicitly with the --help option so I could grep for the specific option I was looking for. When asking explicitly for help docs they should be printed to stdout.
Unix Reader
The Unix Reader is a fascinating read from someone who was there and contributing in the early days of Unix. This document is a good place to start if you enjoy reading about Unix history and origins, or just like to hear some hacker stories.
Leave a comment if you know of other stuff that's printed to stderr besides errors and usage documentation.
Source : http://jstorimer.com/2011/12/29/the-difference-between-stdout-and-stderr.html
