Pipes and Redirection
Every process on Linux is born with three open file descriptors: 0 (stdin), 1 (stdout), and 2 (stderr). By default stdin reads from the keyboard and stdout/stderr write to the terminal, but the shell lets you rewire these streams before a command runs. Redirection sends a stream to or from a file, while a pipe connects the stdout of one process directly to the stdin of another, letting you compose small single-purpose tools into a data-processing pipeline. This design, inherited from Unix, is why chains like cat access.log | grep 500 | wc -l feel natural: each command does one thing well and the shell handles the plumbing.
Cricket analogy: Every bowler has three lines of communication — the umpire (stdin), the scoreboard (stdout), and the third-umpire review (stderr); piping cat access.log | grep 500 | wc -l is like relaying overs from bowler to keeper to scorer in one seamless chain.
Output and input redirection
> redirects stdout to a file, truncating it first; >> appends instead of truncating. < redirects a file's contents into a command's stdin. 2> redirects stderr specifically, and 2>&1 duplicates file descriptor 1 onto descriptor 2 (or vice versa), which is how you merge error output with standard output. Order matters: cmd > file 2>&1 sends stdout to the file, then makes stderr point at wherever stdout currently points (the file); cmd 2>&1 > file merges stderr with the terminal first, then redirects only stdout to the file, leaving stderr on the terminal.
Cricket analogy: cmd > file 2>&1 is like recording the match highlights reel (stdout) to a DVD first, then pointing the umpire's review feed (stderr) at that same DVD, whereas cmd 2>&1 > file sends the review feed live to the stadium screen before highlights are saved.
# Redirect stdout to a file, overwriting it
ls -la /etc > listing.txt
# Append instead of overwrite
echo "backup run at $(date)" >> backup.log
# Redirect stderr only
grep -r "TODO" /usr/src 2> grep-errors.log
# Merge stderr into stdout, then send both to a file
./build.sh > build.log 2>&1
# Discard output entirely
noisy-command > /dev/null 2>&1
# Feed a file as stdin
sort < unsorted-names.txt > sorted-names.txt
# Here-string and here-document
grep "error" <<< "$log_line"
cat <<EOF > report.txt
Generated on $(date)
Host: $(hostname)
EOFPipes and building pipelines
The pipe operator | connects the stdout of the command on its left to the stdin of the command on its right, running both processes concurrently and streaming data between them through a kernel buffer — no temporary file is created. You can chain any number of commands: journalctl -u nginx | grep ERROR | tail -n 20 | tee recent-errors.txt filters and captures the last 20 nginx errors while also printing them. tee is special because it both writes to a file and passes data through to stdout, letting you inspect and persist a stream simultaneously.
Cricket analogy: The pipe journalctl -u nginx | grep ERROR | tail -n 20 | tee recent-errors.txt is like a fielder relaying the ball from deep cover to the keeper to the bowler's end in one throw, while tee is the third umpire who both signals the decision and logs it for replay.
# Classic pipeline: count unique visitor IPs from a web log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Use tee to log and view output at the same time
long-running-job.sh | tee job-output.log
# Pipe into a while-read loop to process each line
grep -l "deprecated" src/*.py | while read -r file; do
echo "Review: $file"
done
# process substitution: treat command output as a file
diff <(sort file1.txt) <(sort file2.txt)PIPESTATUS is a Bash array holding the exit status of every command in the most recent pipeline, e.g. ${PIPESTATUS[0]} for the first command. Since $? after a pipe only reflects the last command's exit code, PIPESTATUS is essential when you need to know if an earlier stage (like grep finding no matches) actually failed. Enabling set -o pipefail makes the whole pipeline's exit status equal to the first non-zero exit code instead of always taking the last command's status.
A pipeline's default exit status is that of its *last* command only, so grep pattern bigfile | sort > out.txt reports success even if grep found zero matches or failed to open the file. In scripts that must catch such failures, add set -o pipefail near the top, and check PIPESTATUS when you need per-stage detail.
Named pipes and file descriptor tricks
Beyond the anonymous pipes created by |, Linux supports named pipes (FIFOs) created with mkfifo, which persist as filesystem entries that any two processes can open independently to communicate, even if they aren't started as part of the same pipeline. You can also open custom file descriptors with exec 3<>somefile for advanced I/O multiplexing, though this is rarely needed outside specialized scripting.
Cricket analogy: A FIFO created with mkfifo is like a designated relay point on the boundary that any two fielders can use to exchange the ball, even without starting the over together; opening a custom file descriptor like exec 3<>somefile is a rare specialist move, like a substitute using an unconventional throwing lane.
# Create and use a named pipe
mkfifo /tmp/mypipe
(echo "hello through the fifo" > /tmp/mypipe &)
cat /tmp/mypipe
rm /tmp/mypipe- File descriptors 0, 1, and 2 correspond to stdin, stdout, and stderr respectively.
>overwrites a file,>>appends;2>&1merges stderr into wherever stdout currently points.- Pipes (
|) stream one command's stdout directly into another's stdin without an intermediate file. - A pipeline's default exit status is the last command's; use
set -o pipefailandPIPESTATUSto catch earlier failures. teeboth writes a stream to a file and passes it through, useful for logging while still viewing live output.- Process substitution
<(...)lets a command's output be read as if it were a file, handy fordiffandcomm.
Practice what you learned
1. Which redirection merges stderr into the same destination as stdout when placed after the stdout redirection?
2. By default, what determines the exit status of a pipeline like `cmd1 | cmd2 | cmd3`?
3. What does the command `sort < names.txt > sorted.txt` do?
4. Which Bash array can you inspect to get the exit status of each individual command in the last executed pipeline?
5. What is the primary advantage of `mkfifo` (a named pipe) over an anonymous shell pipe?
Was this page helpful?
You May Also Like
Writing Your First Bash Script
Get hands-on with the anatomy of a Bash script — shebang, permissions, execution, comments, and structuring commands into a reusable, repeatable tool.
grep and Regular Expressions
Master text searching with grep and the regular expression syntax that powers it, from basic literal matches to extended regex patterns and useful flags.
Exit Codes and Error Handling
Understand how every command reports success or failure through its exit status, and how to build scripts that detect and respond to errors reliably.
Background Jobs and nohup
Learn to run and manage background jobs with &, jobs, bg/fg, and disown, and keep long-running processes alive after logout using nohup and disown.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics