Design an Ad Click Aggregator (High-Level Design)

🔴 High-Level Design · Case Study · Super-senior

"Design an ad-click aggregator" is the quintessential big-data streaming question. Ad platforms need to count clicks — by ad, by minute, by region — at millions of events per second, power real-time dashboards and accurate billing, and never lose or double-count a click (money depends on it). It's less about CRUD and more about stream processing. Let's design it.

Step 1 — Requirements

• Functional: record every ad click; query aggregated counts (clicks per ad per minute/hour, filter by region/device); power a near-real-time dashboard and an accurate billing report.
• Non-functional: very high write throughput (millions/sec), low-latency reads on aggregates, no lost or double-counted clicks, tolerate late-arriving events.

Step 2 — Estimation & the core tension

🟠 At ~1M clicks/sec you cannot INSERT each into a database and COUNT(*) later — the write and query volumes are both impossible that way. The whole design is about aggregating on the fly: turn a firehose of raw events into small, pre-summed counters. The tension to call out: real-time speed vs billing accuracy. Dashboards can tolerate approximate, slightly-stale numbers; billing must be exact. That split drives the architecture.

Step 3 — The pipeline

1. Ingest: click events go into Kafka — a durable, partitioned log that absorbs the firehose and lets you replay if downstream fails.
2. Aggregate: a stream processor (Flink/Spark Streaming) consumes the log and maintains rolling counts in time windows (per ad, per minute).
3. Serve: write rolled-up counts to a fast columnar/OLAP or key-value store the dashboard queries in milliseconds.
4. Reconcile: also archive raw events to a data lake; a nightly batch job recomputes exact totals for billing.

Step 4 — Windowing & late events

🔴 Aggregation happens over tumbling windows (e.g. every 1 minute). Real clicks arrive out of order and late (mobile networks), so use event time (when the click happened) not processing time, plus a watermark — a grace period after which a window is considered closed. Clicks later than the watermark either update a correction or are caught by the nightly batch. Getting event-time windowing right is the senior signal.

Step 5 — Accuracy: exactly-once & dedup

🔴 Because money is involved:

• Idempotency / dedup: each click carries a unique ID; dedupe on it (a Bloom filter or key store) so retries/replays don't double-count.
• Exactly-once processing: use the stream processor's checkpointing + Kafka offsets so a crash-and-replay doesn't re-add counts.
• Lambda architecture: the fast streaming path gives approximate real-time numbers; the batch path gives the authoritative billing total — reconcile the two. (Some teams prefer a single "kappa" stream path with replay for corrections.)
• Hot partitions: a viral ad skews one partition — shard the aggregation key (ad_id + bucket) to spread load, then sum the buckets.

The interview-ready walkthrough

"You can't insert-and-count at millions/sec, so I aggregate in a stream. Clicks land in Kafka (durable, replayable), a stream processor maintains per-ad counts in event-time windows with watermarks for late data, and rolled-up counts go to a fast OLAP store for the dashboard. For billing accuracy I dedupe on click ID, rely on exactly-once checkpointing, and run a nightly batch over archived raw events to reconcile — the lambda pattern. I'd shard hot keys to avoid partition skew." That names the pipeline, windowing, and the accuracy story — exactly the three things asked."

What to read next

• Message Queues & Kafka — the ingestion backbone
• Bloom Filters — cheap dedup at scale
• Design a Leaderboard — next up, another real-time aggregation
• ← The complete System Design guide (hub)

← Design Google Maps · Design a job scheduler →