From Raw Log to Summary
A web server access log is one line per request, with fields for the client IP, timestamp, request line, status code, and bytes sent. AWK is ideal for turning that stream into a summary because it reads each line, splits it into fields, and can accumulate totals in associative arrays keyed by any field. The main block runs once per line to update the counts; the END block runs once after all input, where you emit the final report. This BEGIN / main / END structure is the backbone of nearly every AWK summarizer.
Cricket analogy: Accumulating per-status counts is like a scorer's book tallying runs and wickets ball by ball, then reading out the full scorecard only after the innings closes.
Associative Arrays Do the Work
The core technique is an associative array indexed by a string key. To count requests per status code you write count[$9]++, which creates the entry on first sight and increments it thereafter. To sum bytes per code you write bytes[$9] += $10. Because AWK arrays are hash maps, keys can be IP addresses, URLs, or dates just as easily. In the END block you iterate with for (key in arr) to print each bucket. Note that iteration order is unspecified, so pipe the output through sort if you need it ordered by count or key.
Cricket analogy: count[$9]++ is like a manual clicker counting each boundary a batsman hits; every four adds one to that player's tally in the record.
A Complete Summarizer
#!/usr/bin/awk -f
# loglog.awk — summarize an Apache-style access log
# usage: awk -f loglog.awk access.log | sort -k2 -nr
{
status = $9
reqs[status]++
bytes[status] += $10
total_reqs++
total_bytes += $10
}
END {
print "status requests bytes"
for (s in reqs)
printf "%-6s %8d %10d\n", s, reqs[s], bytes[s]
printf "\nTotal: %d requests, %d bytes across %d status codes\n",
total_reqs, total_bytes, length(reqs)
}printf gives you column alignment that print cannot. The format %-6s left-justifies the status in a 6-wide field, and %8d right-justifies the count in an 8-wide field, so numbers line up in neat columns. length(arr) returns the number of keys in an associative array — a handy way to count distinct values without a manual counter.
Field positions assume a specific log format. The default Apache combined log puts status in $9 and bytes in $10, but a custom LogFormat, extra proxy headers, or spaces inside quoted fields will shift those positions. Always verify field numbers against a real sample line before trusting the summary, or your totals will silently be computed from the wrong columns.
- A log summarizer follows the BEGIN / main / END structure: accumulate per line, report at the end.
- Associative arrays like count[$9]++ and bytes[$9] += $10 group and total by any field key.
- Keys can be IPs, URLs, dates, or status codes since AWK arrays are hash maps.
- Iterate results with for (key in arr); order is unspecified, so pipe through sort if needed.
- printf with width specifiers produces aligned columns; length(arr) counts distinct keys.
- Verify field positions against a real log line, since custom formats shift $9 and $10.
Practice what you learned
1. What does count[$9]++ do the first time a new status code appears?
2. In which block should the final report of a summarizer be printed?
3. Why might you pipe an AWK summarizer's output through sort?
4. What does length(arr) return for an associative array?
5. In a default Apache combined log, which field typically holds the HTTP status code?
Was this page helpful?
You May Also Like
AWK Best Practices
Guidelines for writing AWK programs that are correct, readable, and maintainable, covering quoting, field handling, variable init, and when to move to a script file.
AWK vs Python for Text Processing
A practical comparison of AWK and Python for text and log processing, showing where each tool shines and how to decide between a one-liner and a script.
AWK Quick Reference
A compact reference to AWK's structure, special variables, operators, functions, and the most useful one-liners for everyday text processing.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics