Why Regular Expressions Power sed
sed is a stream editor, and almost everything it does interesting is driven by regular expressions: they decide which lines an address selects and which text a substitution replaces. By default sed uses POSIX Basic Regular Expressions (BRE), a dialect where metacharacters like +, ?, {, }, (, and | are literal unless backslash-escaped. Understanding this dialect is the single biggest lever for writing sed that behaves the way you expect.
Cricket analogy: Just as a batter reads the seam and grip to predict swing before the ball is bowled, you read sed's regex dialect to predict how a pattern will move through the stream before you run it.
Anchors and Character Classes
Anchors pin a match to a position rather than to a character: ^ matches the start of a line and $ matches the end. So s/^#//' strips a leading hash and /^$/ matches empty lines. Character classes inside brackets match any one listed character: [aeiou] matches a single vowel, [0-9] a digit, and [^0-9] any non-digit because a leading caret inside brackets negates the set. POSIX named classes like [[:space:]] and [[:alpha:]] are portable alternatives that survive different locales.
Cricket analogy: An anchor is like the popping crease: ^ and $ don't match a player, they mark a fixed line on the pitch that decides whether a run or a match counts.
Quantifiers and the BRE Escaping Trap
Quantifiers control repetition. In BRE, * (zero-or-more) works unescaped, but + (one-or-more), ? (zero-or-one), and interval {n,m} must be backslash-escaped as \+, \?, and \{n,m\}. Grouping and alternation likewise need escaping: \(...\) and \|. This is the reverse of most modern regex engines and is the number one source of confusion. Passing -E (or -r in GNU sed) switches to Extended Regular Expressions, where +, ?, {}, (), and | are special without backslashes, matching the syntax people already know from grep -E or programming languages.
Cricket analogy: BRE escaping is the reverse-swing of regex: the ball behaves opposite to instinct, so \+ moving away when you expected + to move in catches out even seasoned players.
# BRE (default): quantifiers and groups need backslashes
sed 's/colou\?r/color/g' file.txt # match color OR colour
sed 's/[0-9]\{3\}-[0-9]\{4\}/REDACTED/' file.txt # phone-like pattern
sed -n '/^ERROR\|^WARN/p' log.txt # lines starting ERROR or WARN
# ERE (-E): the same intent without backslash clutter
sed -E 's/colou?r/color/g' file.txt
sed -E 's/[0-9]{3}-[0-9]{4}/REDACTED/' file.txt
sed -En '/^(ERROR|WARN)/p' log.txt
# Backreferences use \1 in BOTH dialects
sed -E 's/(\w+) \1/\1/g' file.txt # collapse a doubled wordGNU sed accepts both -E and -r for Extended Regular Expressions; -E is the POSIX-standard spelling and is more portable to BSD/macOS sed. Prefer -E in scripts you might share.
Backreferences and Greediness
A grouped subexpression captured with \(...\) (BRE) or (...) (ERE) can be recalled in the replacement as \1, \2, and so on, up to \9. The special replacement token & inserts the entire matched text. sed's regex engine is greedy and has no lazy quantifiers, so .* always grabs as much as possible; to match minimally you must constrain the pattern itself, for example using a negated character class like [^"]* instead of .* between quotes. There is also no non-greedy ? modifier as in PCRE.
Cricket analogy: A backreference \1 is a fielder relaying the exact same ball to another position; & is throwing the whole delivery back untouched to the wicketkeeper for a review.
Because sed regexes are always greedy, s/"(.*)"/[\1]/ on a line with two quoted strings captures everything from the first quote to the last. Use s/"([^"]*)"/[\1]/g to match each quoted span separately.
- sed defaults to POSIX Basic Regular Expressions (BRE), where +, ?, {}, (), and | are literal unless backslash-escaped.
- The -E flag (or -r in GNU sed) switches to Extended Regular Expressions, matching the syntax used by grep -E and most languages.
- ^ anchors to line start, $ to line end; [set] matches one listed character and [^set] negates it.
- POSIX classes like [[:digit:]] and [[:space:]] are locale-safe alternatives to ranges like [0-9].
- Groups \(...\)/(...) capture text recalled as \1-\9; & inserts the whole match in the replacement.
- sed's engine is greedy with no lazy quantifiers; constrain patterns with negated classes to match minimally.
Practice what you learned
1. In sed's default (BRE) mode, which pattern matches one or more digits?
2. What does the & symbol represent in a sed replacement?
3. Which flag enables Extended Regular Expressions in a portable, POSIX-standard way?
4. How do you match minimally between two quotes given sed has no lazy quantifiers?
5. What does the regex /^$/ match?
Was this page helpful?
You May Also Like
Line Ranges and Steps
Selecting spans of lines in sed using numeric ranges, the $ last-line symbol, regex-bounded ranges, and GNU's first~step notation for periodic selection.
Negating Addresses
Using the ! operator to invert a sed address so a command runs on every line the address does NOT select, plus common idioms and gotchas.
The Print and Delete Commands
How sed's p (print) and d (delete) commands work, why -n changes everything, and the difference between d and D in the multi-line pattern space.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics