How to Design an Ad Click Aggregator
Learn how to design a real-time ad click aggregator: partitioned streaming ingestion, deduplication, windowed rollups, and reconciliation.
Expected Interview Answer
An ad click aggregator ingests a high-throughput stream of click events, deduplicates and validates them, then rolls them up into per-ad, per-time-window counters using a stream-processing pipeline so advertisers see near-real-time spend and performance metrics.
Clients or ad servers fire click events into a durable log such as Kafka, partitioned by ad ID so all events for one ad land on the same partition for ordered, stateful aggregation. A stream processor (Flink, Kafka Streams, or Spark Streaming) consumes the log, applies windowed aggregation (tumbling or sliding one-minute windows), deduplicates using an idempotency key or bloom filter to filter bot and double-fire clicks, and emits rolled-up counts to a fast key-value or OLAP store like Druid or Cassandra for dashboard queries. Watermarks handle late-arriving events within a bounded lateness window, and exactly-once or at-least-once-with-idempotent-writes semantics keep counts accurate under retries. A separate batch layer periodically reconciles the real-time counts against raw event logs in a data lake to catch and correct any drift, following a lambda-architecture style split between speed and accuracy.
- Gives advertisers near-real-time visibility into spend and click-through performance
- Partitioned, windowed aggregation scales horizontally with click volume
- Deduplication and bot filtering keep billing and metrics trustworthy
- Batch reconciliation catches drift so real-time numbers stay auditable
AI Mentor Explanation
An ad click aggregator is like a stadium’s ball-by-ball scoring pipeline where every delivery is logged the instant it happens, then rolled up into over-by-over and innings totals instead of someone manually re-adding every ball at the end. Scorers on each end feed a shared ledger partitioned by match, so totals for one game never get mixed with another. Duplicate entries, like a no-ball logged twice by two scorers, are caught and reconciled against the official scorebook after the innings. That continuous ingest-then-roll-up pipeline with a later audit pass is exactly how an ad click aggregator turns raw clicks into trustworthy dashboards.
Step-by-Step Explanation
Step 1
Ingest clicks into a partitioned log
Click events are published to Kafka, partitioned by ad ID so all events for an ad are ordered on one partition.
Step 2
Deduplicate and validate
A stream processor filters duplicate or bot-like clicks using an idempotency key or bloom filter before counting.
Step 3
Windowed aggregation
Tumbling or sliding time windows roll clicks up into per-ad, per-minute counters, handling late events via watermarks.
Step 4
Serve and reconcile
Aggregated counts land in a fast store (Druid/Cassandra) for dashboards, while a batch job reconciles against raw logs periodically.
What Interviewer Expects
- Proposes a durable, partitioned event log (Kafka) as the ingestion backbone
- Explains windowed stream aggregation and watermark handling for late events
- Addresses deduplication and bot/fraud click filtering explicitly
- Mentions a reconciliation or lambda-style batch layer for accuracy auditing
Common Mistakes
- Proposing synchronous writes straight to a relational database for every click
- Ignoring deduplication, letting retries or bots inflate advertiser billing
- Forgetting late-arriving events and watermark/lateness handling
- Not separating the fast real-time path from a slower, accurate reconciliation path
Best Answer (HR Friendly)
“An ad click aggregator takes a firehose of click events and turns them into accurate, near-real-time counts advertisers can trust. We stream every click through a durable queue, filter out duplicates or bot traffic, and roll the numbers up in small time windows so dashboards update quickly, then double-check everything later against the full raw log to make sure nothing drifted.”
Code Example
from collections import defaultdict
import time
WINDOW_SECONDS = 60
seen_click_ids = set() # backed by a bloom filter / Redis in production
window_counts = defaultdict(int)
def handle_click_event(event):
click_id = event["click_id"]
if click_id in seen_click_ids:
return # duplicate, drop it
seen_click_ids.add(click_id)
window_start = int(event["timestamp"] // WINDOW_SECONDS) * WINDOW_SECONDS
key = (event["ad_id"], window_start)
window_counts[key] += 1
def flush_window(ad_id, window_start):
key = (ad_id, window_start)
count = window_counts.pop(key, 0)
write_to_store(ad_id=ad_id, window_start=window_start, clicks=count)
def write_to_store(ad_id, window_start, clicks):
# upsert into Druid/Cassandra keyed by (ad_id, window_start)
store.upsert(ad_id, window_start, clicks)Follow-up Questions
- How would you detect and filter bot or fraudulent clicks in the pipeline?
- How do watermarks handle events that arrive after their window has already closed?
- How would you scale the aggregator to handle a 10x spike in click volume during a major campaign?
- What is the lambda architecture and why does a batch reconciliation layer still matter here?
MCQ Practice
1. Why partition the click event log by ad ID rather than randomly?
Partitioning by ad ID keeps all of an ad’s events together and ordered, which is required for correct stateful windowed aggregation.
2. What is the main purpose of watermarks in a windowed streaming aggregator?
Watermarks define an acceptable lateness threshold, letting the system close out a window while still tolerating some delayed events.
3. Why does a lambda-style batch reconciliation layer still matter if streaming aggregation is real-time?
Streaming aggregation trades some accuracy for speed; a periodic batch pass over the complete raw log corrects any drift for auditability.
Flash Cards
Why partition clicks by ad ID? — Keeps all events for one ad ordered on the same partition for correct stateful aggregation.
What deduplicates clicks? — An idempotency key (e.g. click_id) or bloom filter checked before a click is counted.
What is a watermark? — A bound on how long the pipeline waits for late events before finalizing a time window.
Why keep a batch reconciliation layer? — To recompute accurate totals from raw logs and catch drift the real-time path might miss.