The Shape of an AWK Program
Every AWK program is a sequence of pattern { action } rules. For each input record, AWK tests each pattern and runs the action of those that match; a rule with no pattern runs for every record, and a rule with no action prints the record. Two special patterns bracket the run: BEGIN { ... } executes once before any input, and END { ... } executes once after all input. Inside actions you have variables, arithmetic, string functions, control flow, and the field references $1 through $NF plus the whole-record $0.
Cricket analogy: pattern { action } rules are like a fielding plan: for each delivery, whichever condition matches — full toss, edge, yorker — triggers the pre-set response.
Variables and Functions at a Glance
The built-in variables you reach for constantly are NR (record number), NF (field count), FS/OFS (input/output field separators), and RS/ORS (record separators). String functions include length(s), substr(s, m, n), index(s, t), split(s, arr, sep), sub()/gsub() for substitution, and match(s, re). Arithmetic covers +, -, *, /, %, and ** for exponentiation, plus math functions like int(), sqrt(), and sprintf() for formatted strings. Comparison and regex matching use ==, !=, <, and the ~ and !~ operators against /regex/ literals.
Cricket analogy: gsub() replacing all matches is like the third umpire reviewing every frame of an over to correct each no-ball, not just the first.
Everyday One-Liners
# Print columns 1 and 3, tab-separated
awk -F',' 'BEGIN { OFS="\t" } { print $1, $3 }' data.csv
# Print lines where column 5 exceeds 100
awk '$5 > 100' data.txt
# Count occurrences of each value in column 2
awk '{ c[$2]++ } END { for (k in c) print k, c[k] }' data.txt
# Print records matching a regex in $0
awk '/ERROR|WARN/' app.log
# Sum column 4 grouped by column 1
awk '{ sum[$1] += $4 } END { for (k in sum) print k, sum[k] }' data.txt
# Print NR and the line for matches only
awk '/timeout/ { print NR": "$0 }' app.logSet the field separator with -F before the program, e.g. -F',' for CSV or -F'\t' for TSV. For output, set OFS in a BEGIN block. Remember that -F takes a regular expression, so -F'[,;]' splits on either a comma or a semicolon — useful for messy delimiters.
AWK arrays are associative and iteration with for (k in arr) has no guaranteed order. If you print a grouped summary and need it sorted, pipe the output through sort (for example ... | sort -k2 -nr) rather than assuming AWK will emit keys in insertion or numeric order.
- An AWK program is a list of pattern { action } rules run once per record.
- BEGIN runs before input and END after; a bare pattern prints, a bare action runs always.
- Key variables: NR, NF, FS, OFS, RS, ORS, plus fields $1..$NF and $0.
- Core functions: length, substr, index, split, sub/gsub, match, int, sprintf.
- Set input separator with -F (a regex) and output separator via OFS in BEGIN.
- Group and total with associative arrays; pipe to sort for ordered output.
Practice what you learned
1. What does a rule with a pattern but no action do?
2. How do you set the input field separator to a comma when launching AWK?
3. Which function replaces every occurrence of a pattern in a string?
4. Why should grouped output from for (k in arr) often be piped to sort?
5. What does the operator ~ do in AWK?
Was this page helpful?
You May Also Like
AWK Interview Questions
Common AWK interview questions and the concepts behind them, from records and fields to associative arrays, special variables, and one-liner techniques.
AWK Best Practices
Guidelines for writing AWK programs that are correct, readable, and maintainable, covering quoting, field handling, variable init, and when to move to a script file.
Building a Log Summarizer in AWK
A hands-on walkthrough of building a web server log summarizer in AWK using associative arrays, the END block, and formatted output.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics