sed as a Stream Filter
sed is fundamentally a stream editor: it reads input line by line from standard input (or files), applies its editing commands, and writes the result to standard output. This design makes it a natural citizen of Unix pipelines, where the output of one command becomes the input of the next through the pipe operator. Because sed neither buffers the whole file nor requires random access, it slots between producers like cat or curl and consumers like sort or wc without breaking the streaming model.
Cricket analogy: Like a wicketkeeper standing behind the stumps, sed catches every delivery (line) that comes down the pipeline, cleanly gloves or edits it, and relays it on to the fielder (next command) without stopping the over.
Building Multi-Stage Pipelines
Real power comes from chaining sed with other filters. A common pattern is to grep for candidate lines, transform them with sed, and then sort or count the results. Because each stage is a separate process running concurrently, the shell schedules them in parallel and data flows as soon as it is available. You can also place sed in the middle to normalize data before a downstream tool that is picky about format, such as stripping trailing whitespace before a diff or removing comment lines before feeding a config to another parser.
Cricket analogy: Like a well-drilled fielding chain — cover fields it, throws to the bowler's end, who relays to the keeper — each sed stage does one clean job and passes the ball on for the run-out.
# Extract error lines, strip the timestamp prefix, then count unique messages
grep 'ERROR' app.log \
| sed 's/^[0-9-]* [0-9:]* //' \
| sort | uniq -c | sort -rn
# Normalize CSV: trim spaces around commas before feeding to another tool
cat data.csv | sed 's/ *, */,/g' | cut -d, -f2
# Use sed as an in-stream editor with curl (no temp file needed)
curl -s https://example.com/config \
| sed 's/localhost/prod-host/g' \
| tee final.confPrefer piping directly over 'cat file | sed ...'. sed accepts filenames as arguments (sed 's/x/y/' file), which avoids an extra process. The 'cat file | command' pattern is so common it has a nickname: a Useless Use of Cat. Reserve pipes for when the data genuinely comes from another command's output.
Line Buffering and Real-Time Streams
When sed sits in a pipeline reading from a long-running producer like tail -f, output can appear stuck because sed and libc buffer output in large blocks by default. GNU sed offers the -u (or --unbuffered) flag to flush after each line, which is essential for live log monitoring. Without it, you may wait until a 4KB buffer fills before seeing any transformed output, making interactive pipelines feel frozen even though data is flowing correctly.
Cricket analogy: It's like a scoreboard operator batching updates — without unbuffered mode the crowd sees no runs for an over, then a sudden jump; -u posts each single and boundary the instant it happens.
Exit codes propagate oddly in pipelines. By default the shell reports only the last command's exit status, so a failing sed in the middle of a pipe can be masked. Enable 'set -o pipefail' in scripts so any failing stage causes the whole pipeline to report failure, preventing silent data-loss bugs.
- sed reads stdin line by line and writes stdout, making it a natural pipeline filter.
- Chain grep, sed, sort, and uniq to build powerful text-processing one-liners.
- Pass filenames directly to sed rather than using 'cat file | sed' (Useless Use of Cat).
- Use GNU sed's -u/--unbuffered flag for real-time pipelines like 'tail -f | sed'.
- Enable 'set -o pipefail' so a failing sed stage does not get masked by later commands.
- Placing sed mid-pipeline lets you normalize data before format-sensitive downstream tools.
Practice what you learned
1. Why is 'cat file.txt | sed s/a/b/' considered poor practice?
2. Which GNU sed flag is needed for responsive output when reading from 'tail -f'?
3. What does 'set -o pipefail' accomplish in a shell script with pipelines?
4. In the pipeline 'grep ERROR log | sed "s/^[0-9-]* //" | sort | uniq -c', what is sed's role?
Was this page helpful?
You May Also Like
sed vs awk vs grep
A comparison of the three classic Unix text tools — when to reach for grep to search, sed to edit streams, and awk for field-oriented and programmable processing.
Text Transformation Recipes
A practical cookbook of proven sed one-liners for everyday text surgery: deleting lines, extracting ranges, joining lines, and reformatting with capture groups.
Performance and Portability
How to write sed that runs fast on large files and behaves identically across GNU, BSD/macOS, and POSIX sed — covering common divergences and optimization techniques.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics