Monitoring & Observability

Observability is about understanding your system's internal state from external outputs. Mental model: without observability, you're flying blind. With logs you can debug past events. With metrics you see trends. With traces you understand where time goes. Together they form a complete picture. The Three Pillars of Observability Observability is the ability to understand the internal state of a system from its external outputs. The three pillars: Metrics: Numeric measurements over time (CPU %, request rate, error rate). Cheap to store, easy to alert on. Logs: Timestamped records of events (structured JSON preferred). Rich context, expensive at scale. Traces: End-to-end request tracking across services with timing. Essential for microservices. --The Four Golden Signals (Google SRE) Google's Site Reliability Engineering book defines four signals that, if measured well, cover most production issues: Description Time to serve a request Demand on the system Rate of failed requests How full the service is Why these four? They directly answer: is the system slow? under load? failing? about to fall over? --SLI, SLO, and SLA SLI (Service Level Indicator): The actual measurement. "What fraction of requests complete in <200ms?" SLO (Service Level Objective): The target. "99% of requests complete in <200ms over 30 days." SLA (Service Level Agreement): The contractual commitment with consequences. "We guarantee 99.9% availability. Breach = credits." Error budget: If you burn through your error budget, you stop feature development and focus on reliability. --Prometheus Grafana Stack Prometheus scrapes metrics from instrumented services on a pull model. Stores as time-series data. Instrumenting a service (Python): PromQL — querying Prometheus: --ELK Stack: Elasticsearch Logstash Kibana Elasticsearch: Distributed search and analytics engine — stores and indexes logs Logstash: Data ingestion pipeline — collects, transforms, ships logs Kibana: Visualisation layer — dashboards, log exploration, alerting Modern alternative: Elasticsearch Filebeat (lightweight log shipper) Kibana (drop Logstash for simple pipelines) Structured logging (always use JSON): Structured logs are machine-parseable, filterable, and can be correlated with traces via . --Prometheus Data Model and Metric Types Understanding Prometheus's data model is essential for writing effective PromQL queries and designing useful dashboards. Four metric types: Why Histograms over Summaries for latency? Summaries calculate quantiles per instance — you cannot aggregate p99 across 10 servers (p99 of p99 is not p99). Histograms aggregate bucket counts across instances correctly. Always use Histogram for latency, request duration, and payload size. --PromQL: Writing Production-Ready Queries --ELK Stack in Practice The ELK stack (Elasticsearch Logstash Kibana) is the most common log aggregation platform. Understanding its architecture matters for system design interviews. Filebeat (lightweight Go agent) tails log files and ships to Logstash or Elasticsearch directly. One Filebeat per host, minimal CPU/memory footprint. Logstash parses, transforms, and enriches logs. Can parse unstructured nginx logs, add geo-IP data, mask PII, and route to different Elasticsearch indices. Elasticsearch indexes logs for full-text search and aggregation queries. At scale, careful index management is critical — one index per day per service, with Index Lifecycle Management (ILM) to automatically delete old indices.