What is a Write-Ahead Log (WAL) and Why Do Systems Use It?
Learn what a write-ahead log is, why appends are durable-first, and how it enables fast crash recovery in databases.
Expected Interview Answer
A write-ahead log is an append-only file that records every change durably before it is applied to the actual data structures on disk, so a crash can never lose an acknowledged write and recovery simply replays the log.
When a write arrives, the engine first appends a compact record describing the change to the WAL and fsyncs it to durable storage; only after that append succeeds does it acknowledge the write to the caller and later apply the change to the in-memory or on-disk data structures at its own pace. Because the log entry is sequential and append-only, writing to it is fast even on spinning disks, whereas updating the actual indexed data structures (B-trees, hash tables) in place would be slow and random-access heavy. If the process crashes before the in-memory state is flushed, recovery replays the WAL from the last checkpoint to reconstruct exactly the state that was acknowledged, guaranteeing durability and consistency without requiring every write to be a slow random-access disk update. This pattern underlies PostgreSQL, MySQL InnoDB, Kafka partitions and virtually every durable storage engine.
- Guarantees durability — an acknowledged write survives a crash even before applied to the main data structures
- Turns slow random-access writes into fast sequential appends
- Enables fast crash recovery by replaying the log instead of rebuilding from scratch
- Provides the foundation for replication, since the log itself can be shipped to replicas
AI Mentor Explanation
A write-ahead log is like the official scorer jotting every ball onto a paper scoresheet the instant it happens, before updating the big electronic scoreboard. If the scoreboard controller crashes mid-match, the scorer just re-reads the paper sheet from the last confirmed total and rebuilds the display exactly as it was, ball by ball. Writing to the paper sheet is fast because it is just adding the next line; updating the fancy scoreboard display is the slower, heavier operation done afterward. That durable, sequential record-first approach is exactly what a write-ahead log gives a storage engine.
Step-by-Step Explanation
Step 1
Client issues a write
A write request (insert/update/delete) arrives at the storage engine.
Step 2
Append to the WAL and fsync
A compact record describing the change is appended sequentially to the log file and flushed durably to disk.
Step 3
Acknowledge the write
Only after the fsync succeeds does the engine confirm the write to the caller, guaranteeing durability.
Step 4
Apply and checkpoint later
The change is applied to in-memory/on-disk structures asynchronously; periodic checkpoints let old log segments be truncated.
What Interviewer Expects
- Explains that the log is written and fsynced before the change is acknowledged, not after
- Contrasts sequential append performance with slow random-access updates
- Describes crash recovery as replaying the log from the last checkpoint
- Names real systems that use WAL: PostgreSQL, MySQL InnoDB, Kafka
Common Mistakes
- Saying the WAL is written after the data is updated, reversing the actual order
- Confusing a WAL with a general audit log or application log
- Not mentioning checkpointing and log truncation, implying the log grows forever
- Ignoring the fsync/durability guarantee and treating the log as just an optimization
Best Answer (HR Friendly)
“A write-ahead log is a simple idea: before a database changes anything, it first writes down exactly what it is about to do in a fast, append-only file, and only then confirms the write succeeded. If the system crashes, it can replay that file to recover perfectly, so nothing acknowledged is ever lost, and writes stay fast because appending to a log is much cheaper than updating complex data structures directly.”
Code Example
import os
class WriteAheadLog:
def __init__(self, path):
self.fh = open(path, "ab")
def append(self, record: bytes):
length = len(record).to_bytes(4, "big")
self.fh.write(length + record)
self.fh.flush()
os.fsync(self.fh.fileno()) # durable before we return
class Storage:
def __init__(self, wal: WriteAheadLog):
self.wal = wal
self.memtable = {}
def put(self, key, value):
record = f"PUT {key} {value}".encode()
self.wal.append(record) # 1. durable log first
self.memtable[key] = value # 2. apply to memory after
return True # 3. acknowledge only now
def recover(self, records):
for record in records:
op, key, value = record.decode().split(" ", 2)
if op == "PUT":
self.memtable[key] = valueFollow-up Questions
- How does checkpointing bound the size and replay time of a write-ahead log?
- How do databases use the WAL for replication (log shipping) to standby replicas?
- What is the difference between logical and physical WAL records?
- Why does fsyncing the WAL matter for durability, and what happens if it is skipped?
MCQ Practice
1. In the write-ahead logging protocol, when is a write acknowledged to the caller?
WAL requires the log record to be durably persisted first; acknowledgement can happen right after that, before the slower application to main structures.
2. Why is appending to a WAL fast compared to updating a B-tree in place?
Sequential appends are much cheaper than the random-access reads and writes needed to update an indexed data structure in place.
3. What does crash recovery using a WAL typically involve?
Recovery replays WAL entries written since the last checkpoint, restoring exactly the state that had been acknowledged before the crash.
Flash Cards
What is a write-ahead log? — An append-only log that durably records a change before it is applied to the main data structures.
Why is WAL append fast? — It is sequential I/O, unlike the random-access updates needed for indexed data structures.
How does WAL enable crash recovery? — By replaying log records since the last checkpoint to reconstruct exactly the acknowledged state.
Name two real systems using WAL. — PostgreSQL and MySQL InnoDB (also Kafka partition logs).