Design a System for Storing and Querying Logs (Splunk)

To design a Splunk-like system, I would first define the scope: ingesting 50TB daily with sub-second latency for ad-hoc queries. The architecture starts with lightweight agents collecting logs and pushing them to a durable message queue like Kafka to decouple ingestion from processing. Next, we need a distributed storage engine. Instead of row-based storage, I'd implement columnar storage where each field is stored separately. This allows the system to read only the specific columns needed for a query, drastically reducing I/O. For indexing, I would build a global inverted index that maps terms to document IDs and offsets, enabling rapid full-text search. To handle scale, data would be partitioned by timestamp into shards. When a user queries 'error messages from last hour,' the system prunes irrelevant shards immediately. Finally, we ensure durability using replication across availability zones and implement tiered storage, moving cold logs to cheaper object storage while keeping hot data in memory-mapped files for speed. This balances cost, throughput, and query responsiveness effectively.