Design an Image Processing and Filtering Pipeline

To design this pipeline for Meta's scale, I would start by defining the core constraint: we need to process millions of uploads daily while maintaining sub-second latency for immediate previews but allowing async completion for heavy filters. First, upon upload, the service pushes a job message containing the image URI and requested operations into a persistent, partitioned queue like Apache Kafka. This ensures durability; if the consumer crashes, messages remain available. Next, I'd implement a dynamic worker pool using a framework like Celery or Kubernetes Jobs. These workers consume messages from specific partitions. To handle the 'filtering' and 'resizing' logic efficiently, I would use a containerized microservice approach where each worker pulls the image from S3, applies the transformations in memory, and writes results back to a CDN-backed storage bucket. Resilience is critical here. I would implement a Dead Letter Queue (DLQ) for jobs failing after three retries with exponential backoff, triggering an alert for manual review. For persistence, the queue itself acts as the source of truth, but I'd also maintain a status table in DynamoDB to track job states (pending, processing, completed, failed). This allows users to poll their status reliably. Finally, to prevent cascading failures during traffic spikes, I'd integrate auto-scaling policies that spin up new workers based on queue lag metrics, ensuring the system remains responsive even under load.