Design a System for Monitoring E-commerce Price Changes

To design this system, I would start by defining the scope: monitoring thousands of competitors across millions of products with sub-minute latency. The core architecture would consist of a central Scheduler that assigns URL batches to a fleet of Scraper Workers. To address the 'politeness' requirement, each worker implements an adaptive backoff algorithm. Instead of fixed intervals, we use a token bucket approach where the rate limit dynamically adjusts based on HTTP 429 responses from target sites, ensuring we never overwhelm their infrastructure. For efficiency, we won't store every full page. Instead, the Change Detection Engine computes a hash of the relevant price elements. If the new hash matches the previous one, we skip storage entirely. Only when a delta is detected do we write a record containing the timestamp, old price, new price, and currency to a partitioned NoSQL store like Amazon DynamoDB, optimized for time-range queries. This minimizes storage costs and I/O. Finally, we add a notification layer that triggers alerts via SNS when price drops exceed a configurable threshold, enabling rapid repricing decisions. This approach balances high throughput with strict adherence to web ethics.