Design a System for Tracking Daily Active Users (DAU)

To design a system for tracking Daily Active Users at Meta's scale, we first need to define our constraints. We are likely dealing with tens of billions of daily events across Facebook, Instagram, and WhatsApp. Storing every user ID in a set for each day is impossible due to memory constraints. Therefore, we must use a probabilistic approach. I propose an architecture where user activity streams into Kafka topics. A distributed processing layer, perhaps using Flink or Spark Streaming, consumes these events. Instead of storing individual IDs, each node maintains a HyperLogLog sketch for the current day. When a user event arrives, we hash their ID and update the corresponding HLL register. The key advantage here is space efficiency; an HLL sketch might only take 10KB per shard regardless of whether we track a million or a billion users. To get the global DAU, we don't sum the estimates directly. Instead, we perform a bitwise OR operation on all the sketches from different shards to merge them into a single master sketch, then apply the cardinality estimation formula. We must also address the definition of a 'day'. We should partition data by UTC time or a specific region to handle timezone discrepancies accurately. Finally, while HLL introduces a small error margin, typically around 1-2%, we can validate this periodically by sampling a subset of traffic and comparing it against an exact counting method to ensure our estimates remain trustworthy for business decisions.