Discuss Database Sharding Strategies

Database sharding is the process of horizontally partitioning a large dataset across multiple servers to overcome the storage and compute limits of a single machine. In a context like Meta's, where we manage billions of daily active users, a monolithic database becomes a bottleneck, making sharding essential for scalability. First, we distinguish between vertical and horizontal sharding. Vertical sharding splits tables by column, which helps if different features have distinct access patterns, but it doesn't solve write throughput issues. Horizontal sharding, which splits rows across nodes, is our primary strategy for scaling writes and reads. Choosing the right sharding key is critical. A hash-based approach uses a hash function on the key to distribute data evenly, preventing hotspots but making range queries difficult. Conversely, range-based sharding groups data by value ranges, which is excellent for time-series data or geospatial queries, but risks uneven load if data distribution isn't uniform. For complex scenarios, a directory-based approach maintains a mapping service, offering flexibility but introducing a single point of failure or latency. The most challenging aspect is re-sharding. As data grows, we must add nodes. This triggers a data migration process where we must balance traffic, ensure eventual consistency during the move, and avoid locking the entire system. At Meta, we often use consistent hashing rings to minimize data movement during these transitions, ensuring that only a small fraction of keys need to be relocated rather than the entire dataset.