Design a System for A/B Testing

To design an A/B testing system, I'd start by clarifying that we need to support millions of daily active users with sub-10ms latency for flag resolution. The core component is the Bucketing Service. When a request arrives, we extract the user ID and the experiment key. We apply a deterministic hash function, such as MurmurHash3, combining the user ID with the experiment salt. This ensures that a specific user always lands in the same bucket, say Variant A or B, regardless of which server handles the request. For state persistence, we cannot rely on local memory; we must use a highly available distributed store like Google Spanner or a sharded Redis cluster to store the current experiment configuration and traffic weights. If we change a weight from 50/50 to 90/10, the service must immediately reflect this without restarting. To handle analysis, every exposure event is logged asynchronously to a message queue like Pub/Sub, then aggregated into BigQuery. This allows data scientists to run statistical significance tests later. Crucially, we must implement a fallback mechanism: if the bucketing service is down, we default to the control group to prevent breaking the application. Finally, we need a robust dashboard for product managers to visualize conversion rates and stop experiments instantly if negative impacts are detected.