Design a System for Distributed Tracing (Jaeger/Zipkin)

To design a distributed tracing system like Jaeger for Google-scale services, I would start by establishing the core components: instrumentation agents, collectors, and a storage backend. First, each microservice generates spans representing operations, propagating trace IDs using standard headers to maintain context across boundaries. These spans are sent asynchronously to local sidecar agents to prevent blocking the main application thread. Next, the agents batch and forward data to a centralized collector cluster. A critical challenge here is volume; sending every single request is unsustainable. I would implement adaptive sampling. Head-based sampling would randomly sample 1% of traffic to get a statistical overview. However, for debugging, we need tail-based sampling. If any span in a trace exceeds a latency threshold or returns an error, the entire trace is sampled and stored. This ensures we capture failures without overwhelming our storage. For storage, I'd recommend a time-series optimized database like Bigtable or Cassandra, partitioned by trace ID to ensure fast retrieval. The query layer would allow engineers to filter by service name, operation duration, or error codes. Finally, visualization tools would aggregate these spans into a Gantt-chart style timeline, highlighting bottlenecks where latency spikes occur between specific service calls, enabling rapid root cause analysis during outages.