Regular Expressions in sed

Analogy categoryAnalogies

Why Regular Expressions Power sed

sed is a stream editor, and almost everything it does interesting is driven by regular expressions: they decide which lines an address selects and which text a substitution replaces. By default sed uses POSIX Basic Regular Expressions (BRE), a dialect where metacharacters like +, ?, {, }, (, and | are literal unless backslash-escaped. Understanding this dialect is the single biggest lever for writing sed that behaves the way you expect.

🏏

Cricket analogy: Just as a batter reads the seam and grip to predict swing before the ball is bowled, you read sed's regex dialect to predict how a pattern will move through the stream before you run it.

Anchors and Character Classes

Anchors pin a match to a position rather than to a character: ^ matches the start of a line and $ matches the end. So s/^#//' strips a leading hash and /^$/ matches empty lines. Character classes inside brackets match any one listed character: [aeiou] matches a single vowel, [0-9] a digit, and [^0-9] any non-digit because a leading caret inside brackets negates the set. POSIX named classes like [[:space:]] and [[:alpha:]] are portable alternatives that survive different locales.

🏏

Cricket analogy: An anchor is like the popping crease: ^ and $ don't match a player, they mark a fixed line on the pitch that decides whether a run or a match counts.

Quantifiers and the BRE Escaping Trap

Quantifiers control repetition. In BRE, * (zero-or-more) works unescaped, but + (one-or-more), ? (zero-or-one), and interval {n,m} must be backslash-escaped as \+, \?, and \{n,m\}. Grouping and alternation likewise need escaping: $...$ and \|. This is the reverse of most modern regex engines and is the number one source of confusion. Passing -E (or -r in GNU sed) switches to Extended Regular Expressions, where +, ?, {}, (), and | are special without backslashes, matching the syntax people already know from grep -E or programming languages.

🏏

Cricket analogy: BRE escaping is the reverse-swing of regex: the ball behaves opposite to instinct, so \+ moving away when you expected + to move in catches out even seasoned players.

bash

# BRE (default): quantifiers and groups need backslashes
sed 's/colou\?r/color/g' file.txt          # match color OR colour
sed 's/[0-9]\{3\}-[0-9]\{4\}/REDACTED/' file.txt  # phone-like pattern
sed -n '/^ERROR\|^WARN/p' log.txt          # lines starting ERROR or WARN

# ERE (-E): the same intent without backslash clutter
sed -E 's/colou?r/color/g' file.txt
sed -E 's/[0-9]{3}-[0-9]{4}/REDACTED/' file.txt
sed -En '/^(ERROR|WARN)/p' log.txt

# Backreferences use \1 in BOTH dialects
sed -E 's/(\w+) \1/\1/g' file.txt          # collapse a doubled word

GNU sed accepts both -E and -r for Extended Regular Expressions; -E is the POSIX-standard spelling and is more portable to BSD/macOS sed. Prefer -E in scripts you might share.

Backreferences and Greediness

A grouped subexpression captured with $...$ (BRE) or (...) (ERE) can be recalled in the replacement as \1, \2, and so on, up to \9. The special replacement token & inserts the entire matched text. sed's regex engine is greedy and has no lazy quantifiers, so .* always grabs as much as possible; to match minimally you must constrain the pattern itself, for example using a negated character class like [^"]* instead of .* between quotes. There is also no non-greedy ? modifier as in PCRE.

🏏

Cricket analogy: A backreference \1 is a fielder relaying the exact same ball to another position; & is throwing the whole delivery back untouched to the wicketkeeper for a review.

Because sed regexes are always greedy, s/"(.*)"/[\1]/ on a line with two quoted strings captures everything from the first quote to the last. Use s/"([^"]*)"/[\1]/g to match each quoted span separately.

sed defaults to POSIX Basic Regular Expressions (BRE), where +, ?, {}, (), and | are literal unless backslash-escaped.
The -E flag (or -r in GNU sed) switches to Extended Regular Expressions, matching the syntax used by grep -E and most languages.
^ anchors to line start, $ to line end; [set] matches one listed character and [^set] negates it.
POSIX classes like [[:digit:]] and [[:space:]] are locale-safe alternatives to ranges like [0-9].
Groups $...$/(...) capture text recalled as \1-\9; & inserts the whole match in the replacement.
sed's engine is greedy with no lazy quantifiers; constrain patterns with negated classes to match minimally.