Sorting and Cutting Text with sort, cut, uniq
Many real shell tasks — finding the most frequent IP address in a log, extracting a single column from a CSV, or listing unique values in a field — don't need a full programming language like awk; they're solved faster and more readably by chaining three small, single-purpose tools: sort to reorder lines, cut to slice out columns, and uniq to collapse or count adjacent duplicate lines. Understanding how these three compose is one of the highest-leverage skills in everyday shell work, embodying the Unix philosophy of small tools joined by pipes.
Cricket analogy: Finding the most-dismissed batsman in a season doesn't need a full statistics package — chaining sort to order deliveries, cut to slice out the batsman column, and uniq to count repeats is the small-tools Unix philosophy, like a scorer using three simple ledgers instead of one giant spreadsheet.
sort: ordering lines
sort orders input lines, alphabetically by default. -n switches to numeric comparison (essential — without it, '10' sorts before '9' because plain sort compares character by character); -r reverses the order; -k selects which field to sort by (e.g. -k2,2 sorts by the second field only); and -u removes duplicate lines as part of sorting, similar to piping through uniq afterward but slightly more efficient since it doesn't require a separate pass. -t sets the field delimiter for -k, since sort's default field splitting is whitespace, much like awk's. Combining -k with a numeric suffix like -k5,5n sorts specifically by field 5 as a number, which is common when sorting du or ps output by size or CPU.
Cricket analogy: sort -n on over counts avoids '10' sorting before '9' as plain text would; -r reverses to show the highest wicket-takers first, -k2,2 sorts by the batsman column, -t, sets a comma delimiter, and -k5,5n sorts a CSV of strike rates numerically by the fifth field, like sorting ps output by CPU.
# Numeric sort (without -n, '10' would sort before '9')
sort -n numbers.txt
# Reverse numeric sort: largest first
sort -rn scores.txt
# Sort by the 3rd colon-delimited field, numerically
sort -t: -k3,3n /etc/passwd
# Sort du output by size (human-readable, so use -h not -n)
du -sh */ | sort -rh | head -10
# cut: extract fields 1 and 3 from a colon-delimited file
cut -d: -f1,3 /etc/passwd
# cut: extract a fixed character range (e.g. columns 1-10)
cut -c1-10 fixedwidth.txt
# uniq: count occurrences of each unique line (input MUST be sorted first)
sort access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -5
# uniq: show only lines that appear more than once
sort emails.txt | uniq -dcut and uniq: slicing and collapsing
cut extracts portions of each line, either by delimiter-separated field number (-d sets the delimiter, -f selects fields) or by fixed character position (-c). Unlike awk, cut has no concept of collapsing repeated delimiters — cut -d' ' -f1 on text with multiple consecutive spaces produces empty fields, which is often the reason people reach for awk instead when dealing with irregularly-spaced command output. uniq collapses consecutive identical lines into one, but critically it only compares adjacent lines — non-adjacent duplicates are left alone entirely. This is why uniq is almost always preceded by sort in a pipeline: sorting guarantees identical lines become adjacent, so uniq (and its -c count flag, or -d/-u to show only duplicated/unique lines) actually works as intended.
Cricket analogy: cut -d' ' -f1 on a scorecard with irregular spacing between overs produces empty fields since cut doesn't collapse repeated delimiters like awk would; and uniq -c only counts adjacent identical entries, which is why you must sort the list of dismissed batsmen first so repeats become adjacent before counting.
The classic pattern sort | uniq -c | sort -rn — sort to group duplicates, uniq -c to count them, sort -rn to rank by frequency — is one of the most reused idioms in shell scripting, appearing everywhere from analyzing web server logs for top requesters to counting word frequency in text corpora. It's worth committing to memory.
Running uniq on unsorted input silently produces wrong results without any error — it will only merge duplicates that happen to be adjacent, leaving every other repeated occurrence untouched and uncounted. Always sort before uniq unless you specifically know the input is already grouped.
- sort defaults to lexical ordering; use -n for numeric, -r to reverse, -k to sort by a specific field, -t to set the field delimiter.
- cut extracts fields with -d (delimiter) and -f (field list), or fixed character ranges with -c; it does not collapse repeated delimiters like awk does.
- uniq only removes/counts ADJACENT duplicate lines, so input must be sorted first for correct deduplication or counting.
- The idiom
sort | uniq -c | sort -rncounts occurrences and ranks them from most to least frequent — extremely common in log analysis. - uniq -d shows only lines that had duplicates; uniq -u shows only lines that appeared exactly once.
- For irregularly whitespace-separated data, prefer awk's field splitting over cut, which treats every delimiter occurrence literally.
Practice what you learned
1. Why does plain `sort` (without -n) order '9' after '10' when sorting a list of numbers as text?
2. Why does `uniq` need its input to be pre-sorted to correctly count all duplicate lines?
3. What does `cut -d: -f1,3 /etc/passwd` extract?
4. What does the pipeline `sort access.log | awk '{print $1}' | sort | uniq -c | sort -rn` most likely compute?
5. What is the key limitation of `cut` compared to `awk` when splitting on whitespace?
Was this page helpful?
You May Also Like
awk Basics
Learn awk, the field-oriented text processing language, to extract columns, compute aggregates, and transform structured text like logs and CSVs.
grep and Regular Expressions
Master text searching with grep and the regular expression syntax that powers it, from basic literal matches to extended regex patterns and useful flags.
Pipes and Redirection
Learn how the shell wires standard input, output, and error between commands and files using redirection operators and pipes to build powerful command chains.
Disk Usage and Management (df, du, mount)
Learn how to inspect filesystem capacity, measure directory sizes, and understand how block devices get attached to the Linux directory tree via mounting.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics