awk Basics
awk is a full pattern-scanning and text-processing language, named after its creators Aho, Weinberg, and Kernighan. Where grep finds lines and sed transforms text, awk's superpower is treating each input line as a record automatically split into fields, which makes it exceptionally good at working with column-oriented data: log files, CSVs, ps or df output, and any whitespace- or delimiter-separated text. A typical awk program is a series of pattern { action } pairs — for every input line, awk tests each pattern and runs the corresponding action if it matches, with an empty pattern matching every line.
Cricket analogy: awk treats each line of a scorecard as a full record automatically split into fields (batsman, runs, balls), unlike grep which just finds lines mentioning 'century'; a pattern like /century/ {print} acts on matching records the way a commentator flags a milestone.
Fields, records, and built-in variables
By default awk splits each line (a 'record') on runs of whitespace into fields referenced as $1, $2, and so on, with $0 representing the entire line. NF (Number of Fields) holds the count of fields in the current record, so $NF conveniently refers to the last field regardless of how many there are. NR (Number of Records) is a running counter of lines processed so far, useful for numbering output or skipping headers. The field separator defaults to whitespace but is changed with -F, e.g. -F, for comma-separated data, or by setting the FS variable inside a BEGIN block. BEGIN and END blocks run once before any input is read and once after all input has been processed respectively, making them the natural place to initialize variables or print summary totals.
Cricket analogy: In a ball-by-ball CSV, $1 is the over, $NF is always the runs scored on that ball regardless of how many fields precede it, and NR counts deliveries bowled so far; switching -F, handles comma-separated scorecards, and a BEGIN block can initialize a running total.
# Print the 1st and 3rd whitespace-separated fields of every line
awk '{print $1, $3}' access.log
# Comma-separated input: print name and email columns
awk -F, '{print $2, $4}' users.csv
# Only print lines where the 5th field (status code) is >= 500
awk '$5 >= 500 {print $0}' access.log
# Sum a numeric column (e.g. bytes transferred, field 10) and print total at end
awk '{sum += $10} END {print "Total bytes:", sum}' access.log
# Print line count, number fields, and last field per line
awk '{print NR": "NF" fields, last="$NF}' data.txt
# Combine grep-like filtering with computed output: average response time by status code
awk '{count[$5]++; total[$5]+=$NF} END {for (c in count) printf "%s: avg=%.2f n=%d\n", c, total[c]/count[c], count[c]}' access.logPatterns, associative arrays, and printf
A pattern before an action can be a regex (/ERROR/ {print}), a comparison ($3 > 100), a range, or combinations with &&/||. Omitting the action defaults to {print $0}, so awk '/ERROR/' app.log behaves much like grep ERROR app.log. awk's arrays are associative (keyed by string, similar to a hash map/dictionary) rather than strictly numeric-indexed, which is what makes the counting and grouping idiom count[$1]++ so natural — it works whether $1 is a number, a username, or an IP address. printf gives formatted output control (%s, %d, %.2f, field widths) that plain print lacks, which matters when producing aligned reports or fixed-decimal numbers.
Cricket analogy: /ERROR/ {print} in awk is like a scout flagging every scorecard line mentioning 'dismissed', similarly awk '/ERROR/' app.log behaves like grep ERROR app.log; count[$1]++ tallies dismissals per bowler the way an associative array groups wickets by bowler name.
The 'one-liner' awk '{print $1}' file and the equivalent cut -d' ' -f1 file might look interchangeable, but awk's field splitting collapses runs of whitespace by default (so multiple spaces or tabs between fields don't produce empty fields), while cut splits on the literal delimiter character every time, producing empty fields for consecutive delimiters. This is why awk is often preferred for irregularly-spaced command output like ps or df -h.
Forgetting that awk's default field separator is 'any run of whitespace,' not literally a single space, trips people up when they explicitly set -F' '(a single space) expecting the same behavior — that setting switches to splitting on each individual space character, producing empty fields wherever there are multiple consecutive spaces. Leave FS at its default for whitespace-delimited data instead of setting it explicitly to a single space.
- awk programs are pattern { action } pairs; an empty pattern matches every line, and an omitted action defaults to printing the line.
- $0 is the whole line, $1..$N are fields, NF is field count, NR is the current record (line) number.
- -F sets the field separator (e.g. -F, for CSV); BEGIN/END blocks run once before/after processing all input.
- Associative arrays (arr[key]++ or arr[key]+=val) make counting and grouping by a field a one-liner.
- printf gives formatted output (%s, %d, %.2f) beyond what plain print offers.
- awk's default whitespace splitting collapses consecutive spaces/tabs, unlike cut which treats each delimiter occurrence literally.
Practice what you learned
1. In an awk program processing a log file, what does `$NF` refer to?
2. What is the purpose of an awk BEGIN block?
3. Why is `awk '{count[$1]++} END {print count["foo"]}'` possible without declaring or sizing the array in advance?
4. What key difference distinguishes awk's default field splitting from `cut -d' '`?
5. Which awk pattern behaves most like `grep ERROR app.log`?
Was this page helpful?
You May Also Like
grep and Regular Expressions
Master text searching with grep and the regular expression syntax that powers it, from basic literal matches to extended regex patterns and useful flags.
sed: Stream Editing Basics
Learn sed, the non-interactive stream editor, for search-and-replace, line deletion, and text transformation directly from the command line or in scripts.
Sorting and Cutting Text with sort, cut, uniq
Combine sort, cut, and uniq to reorder lines, extract specific columns, and deduplicate or count repeated entries in text data pipelines.
Pipes and Redirection
Learn how the shell wires standard input, output, and error between commands and files using redirection operators and pipes to build powerful command chains.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics