How to Design a Real-Time Analytics Dashboard?
Learn how to design a real-time analytics dashboard: streaming rollups, WebSocket push updates, and separate OLAP historical queries.
Expected Interview Answer
A real-time analytics dashboard is designed around a streaming ingestion pipeline that continuously aggregates raw events into pre-computed rollups at multiple time granularities, pushing incremental updates to connected clients over WebSockets, so the dashboard never has to run an expensive query over raw event data to render or refresh.
Raw events (page views, transactions, sensor readings) are ingested into a durable log (Kafka/Kinesis), then consumed by a stream processor (Flink, Kafka Streams, or a windowed aggregation job) that maintains rolling counts, sums, and percentiles in fixed time windows (last minute, last hour, last day) stored in a fast time-series or in-memory store. The dashboard’s API layer reads only from these pre-aggregated rollups, never scanning raw events, keeping query latency flat regardless of total event volume. For the “live” feel, the backend pushes incremental deltas to subscribed dashboard clients over WebSockets as new aggregation windows close, rather than requiring the client to poll. Historical/drill-down queries (e.g., "show me last month broken down by day") are served from a separate OLAP store (ClickHouse, Druid, or a data warehouse) optimized for that access pattern, distinct from the hot real-time rollups. At scale, ingestion and aggregation are partitioned by a natural key (tenant ID, metric type) so one noisy source does not delay aggregation for everyone else, and late-arriving events are handled via watermarking so windows close correctly without waiting forever for stragglers.
- Pre-aggregated rollups keep dashboard query latency constant regardless of raw event volume
- WebSocket push delivers a genuinely live feel without client polling overhead
- Separating hot rollups from an OLAP store lets both real-time and historical drill-down queries be fast
- Partitioning by tenant/metric isolates noisy sources so they cannot delay everyone else’s dashboard
AI Mentor Explanation
A real-time analytics dashboard is like the live scoreboard that updates a batting average and run-rate the instant a ball is bowled, without anyone recalculating the whole match from ball one each time. A scorer keeps a running tally per over and per innings, and the board just adds the newest delta rather than replaying the entire match history. If someone wants last season’s stats broken down by match, that comes from a separate, slower archive built for deep digging, not from the live tally system. That split between instantly updated rolling stats and a separate historical archive is exactly how a real-time analytics dashboard is built.
Step-by-Step Explanation
Step 1
Ingest events into a durable stream
Raw events are published to a log (Kafka/Kinesis) that decouples producers from downstream aggregation and survives consumer restarts.
Step 2
Aggregate with windowed stream processing
A stream processor maintains rolling counts, sums, and percentiles per fixed time window, handling late-arriving events via watermarking.
Step 3
Serve the dashboard from pre-aggregated rollups
The API layer reads only from the fast rollup store, never scanning raw events, so query latency stays flat as volume grows.
Step 4
Push live deltas and route historical queries separately
New window closes are pushed to clients over WebSockets; drill-down/historical queries go to a separate OLAP store optimized for that access pattern.
What Interviewer Expects
- Distinguishes the hot streaming-aggregation path from historical/drill-down OLAP queries
- Explains windowed aggregation with pre-computed rollups instead of scanning raw events per request
- Mentions push-based delivery (WebSockets) for the “live” feel instead of client polling
- Addresses late-arriving events (watermarking) and partitioning for isolation at scale
Common Mistakes
- Querying raw event tables directly for every dashboard refresh, causing latency to grow with data volume
- Using client-side polling instead of push-based updates for a dashboard branded as “real-time”
- Ignoring late-arriving/out-of-order events, causing incorrect or unstable window aggregates
- Not separating hot real-time rollups from historical analytics, forcing one store to serve both patterns poorly
Best Answer (HR Friendly)
“A real-time analytics dashboard needs to feel instantly up to date without recalculating everything from scratch on every refresh, so I would stream events into rolling aggregates that update incrementally, and push those updates straight to the dashboard instead of making it repeatedly ask for new data. For digging into older history, I would use a separate system built for that kind of deep, slower query so it does not slow down the live view.”
Code Example
pipeline:
source:
type: kafka
topic: raw-events
partitionKey: tenantId
windowedAggregation:
windows:
- size: 1m
watermarkDelay: 10s
- size: 1h
watermarkDelay: 2m
metrics:
- name: eventCount
op: count
- name: p95Latency
op: percentile
percentile: 95
sink:
rollupStore:
type: redis-timeseries
livePush:
type: websocket
topicPattern: "dashboard:{tenantId}"
historicalStore:
type: clickhouse
retentionDays: 400Follow-up Questions
- How would you handle a burst of late-arriving events that should belong to an already-closed time window?
- How would you keep dashboard queries fast for a tenant with orders of magnitude more events than others?
- How would you design exactly-once aggregation semantics if the stream processor can restart mid-window?
- How would you scale WebSocket fan-out to millions of dashboard viewers watching different metrics?
MCQ Practice
1. Why does a real-time analytics dashboard read from pre-aggregated rollups instead of scanning raw events on every request?
Scanning raw events per request would make latency grow with data volume; pre-aggregated rollups keep response time constant.
2. What is watermarking used for in a windowed streaming aggregation pipeline?
Watermarks let the stream processor decide when it is safe to finalize a window, tolerating some lateness without waiting indefinitely.
3. Why is a separate OLAP store typically used for historical drill-down queries instead of the real-time rollup store?
Real-time rollups are optimized for recent, fixed-window access; OLAP stores like ClickHouse/Druid are built for flexible historical aggregation.
Flash Cards
Why use pre-aggregated rollups for the dashboard? — So query latency stays flat regardless of total raw event volume, instead of growing as data accumulates.
How is the “live” feel achieved? — By pushing incremental deltas to dashboard clients over WebSockets as aggregation windows close, instead of client polling.
What is watermarking for? — Deciding when a time window can be safely closed despite some events arriving late, without waiting forever.
Why separate real-time rollups from an OLAP store? — Real-time rollups serve fast recent-window reads; OLAP stores serve flexible, deep historical drill-down queries — each is optimized differently.