Design a Distributed File System (HDFS/S3)

To design a robust Distributed File System like HDFS or S3, we start by defining the primary goal: storing petabytes of data with high durability even when hardware fails. I propose a Master-Slave architecture. The Master, or NameNode, holds all namespace metadata and manages the mapping of file paths to data blocks. It does not store the actual data itself. The slaves, or DataNodes, store the physical blocks of data and handle read/write requests. For scalability, we split large files into fixed-size blocks, typically 128MB. This allows parallel processing across many nodes. To ensure fault tolerance, every block is replicated three times across different racks. If a DataNode fails, its heartbeat stops reaching the NameNode. The Master immediately triggers a re-replication process using other surviving replicas to maintain the desired replication factor. This ensures no data loss during failures. We also implement lease mechanisms for write operations to prevent concurrent modifications. For Amazon specifically, this aligns with their principle of 'Customer Obsession' by ensuring data integrity and availability, which are critical for services like AWS S3. Finally, we consider rack awareness to minimize network traffic and maximize resilience against rack-level outages.