awk as a Programming Language
awk is not just a field-extraction tool; it is a complete pattern-action programming language with variables, associative arrays, control flow, and user-definable functions, structured as a series of pattern { action } rules that run once per input record. Every rule's pattern is tested against each line, and any rule whose pattern matches (or has no pattern, meaning always match) has its action block executed, which is why awk one-liners like awk '/ERROR/ {print $0}' work without any explicit loop.
Cricket analogy: A DRS review system that automatically triggers a specific replay routine only when a specific condition (like an lbw appeal) occurs is like an awk pattern only running its action block when a line matches that pattern.
Associative Arrays and Field Separators
awk's arrays are associative by default, so count[$1]++ creates an entry keyed by the literal string value of field one, letting you tally occurrences, group records, or build lookup tables without declaring a fixed size. Field separation is controlled by -F or the FS variable, which accepts not just single characters but full regular expressions (e.g. -F'[,:]' splits on either a comma or a colon), and OFS controls what separator is used when a modified record is rebuilt with print $0 after any field has been reassigned.
Cricket analogy: A scorer tallying how many boundaries each batter has hit using a dictionary keyed by the batter's name, rather than a fixed-size array by batting order, mirrors awk's associative array count[$1]++ keyed by an arbitrary field value.
# Tally page views per URL from an access log, splitting on multiple delimiters
awk -F'[ \t]+' '{ count[$7]++ } END { for (url in count) print count[url], url }' access.log | sort -rn
# User-defined function to compute a percentage, called from the main rules
awk '
function pct(part, whole) { return (whole == 0) ? 0 : (part / whole) * 100 }
BEGIN { FS = "," }
{ total += $2; sums[$1] += $2 }
END {
for (name in sums)
printf "%-10s %6.2f%%\n", name, pct(sums[name], total)
}' sales.csvBEGIN, END, and User-Defined Functions
The special BEGIN block runs exactly once before any input is read, making it the right place to set FS/OFS, print a header, or initialize arrays, while END runs exactly once after all input has been processed, ideal for printing totals or iterating over an accumulated array. awk also supports user-defined functions declared with the function keyword, which can take parameters and use return; because awk passes arrays to functions by reference but scalars by value, a function can mutate a passed-in array in place, which is a common technique for building reusable aggregation helpers.
Cricket analogy: A pre-match pitch inspection that happens exactly once before play begins, and a post-match presentation ceremony that happens exactly once after the final ball, mirror awk's BEGIN and END blocks running exactly once each.
In awk, arrays are passed to functions by reference (mutations inside the function persist outside it), while scalars are passed by value (a copy). This lets you write a function like function tally(arr, key) { arr[key]++ } and call tally(counts, $1) to update the caller's counts array directly.
- awk programs are sequences of
pattern { action }rules, run once per input record. - Associative arrays like
count[$1]++let you tally or group data keyed by any string value, not just numeric indices. -ForFScan be a full regular expression, enabling splitting on multiple or complex delimiters.OFScontrols the separator used when a modified$0is rebuilt after reassigning a field.BEGINruns exactly once before input processing;ENDruns exactly once after all records are read.- User-defined functions use the
functionkeyword, supportreturn, and pass arrays by reference but scalars by value. - Combining BEGIN/END with associative arrays is the standard pattern for aggregation reports (sums, counts, top-N).
Practice what you learned
1. In awk, when does the BEGIN block execute?
2. What kind of arrays does awk provide by default?
3. How are arrays passed to user-defined functions in awk?
4. What does `awk -F'[,:]' '{print $2}' file` do?
5. What controls the separator used when awk rebuilds $0 after a field like $2 has been reassigned?
Was this page helpful?
You May Also Like
Advanced sed Scripting
Move past basic search-and-replace to use sed's address ranges, hold space, branching, and in-place editing for multi-line, stateful text transformations.
Combining grep, awk, and sed
Learn how to chain grep, awk, and sed into efficient pipelines, choosing the right tool for filtering, transforming, and extracting text at each stage.
Advanced Redirection
Master file descriptor manipulation, combined stdout/stderr redirection, process substitution, and safety guards so a script's input and output flow exactly where you intend.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics