Application Observability: Logs, Metrics, and Traces

Monitoring vs observability

These words get used interchangeably, but the distinction is real and useful. Monitoring answers questions you already knew to ask: is CPU high? is the error rate above threshold? It's dashboards and alerts for known failure modes. Observability is the property of being able to ask *new* questions about your system from the outside — to debug behavior you didn't anticipate, without shipping new code.

You get observability by emitting rich, correlated telemetry. The classic model is three pillars: logs, metrics, and traces. Each answers a different question.

The three pillars

Metrics

Metrics are numeric measurements over time: request rate, error rate, latency percentiles, queue depth, memory usage. They're cheap to store because they're aggregated, and they're what you alert on. The canonical method for service-level metrics is RED (Rate, Errors, Duration) for request-driven services, and USE (Utilization, Saturation, Errors) for resources.

The one mistake everyone makes: alerting on averages. Average latency hides the tail. Always track percentiles — p50, p95, p99 — because your unhappy users live in p99.

Logs

Logs are discrete, timestamped events. The single biggest upgrade you can make is structured logging — emit JSON, not free text — so you can filter and aggregate.

{
  "ts": "2026-02-13T10:14:22Z",
  "level": "error",
  "service": "checkout",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "user_id": "u_1029",
  "msg": "payment provider timeout",
  "provider": "stripe",
  "latency_ms": 30021
}

That trace_id field is the magic — it's how you jump from a log line to the full distributed trace.

Traces

In a microservices or even a modestly distributed app, a single user request fans out across services. A distributed trace stitches those hops into one causal timeline. Each unit of work is a span; spans share a trace ID and nest under a root span. When latency spikes, a trace shows you *which* hop ate the time — the database call, the third-party API, the queue wait.

Tie them together with correlation IDs

The pillars are exponentially more useful when correlated. The pattern:

1. 1At the edge (your ingress/gateway), generate a trace ID for each incoming request.
2. 2Propagate it downstream via headers (the W3C traceparent header is the standard).
3. 3Include the trace ID in every log line.
4. 4Now: alert fires on a metric → pivot to traces for that window → find the slow span → jump to the exact logs for that trace.

This "metrics → traces → logs" drill-down is the whole game. Without correlation IDs, you're grepping blind.

OpenTelemetry: the vendor-neutral standard

The industry has converged on OpenTelemetry (OTel) — a CNCF project providing a single set of APIs, SDKs, and a wire protocol (OTLP) for all three signals. The win is decoupling: instrument your code once against OTel, then send telemetry to *any* backend (Prometheus, Jaeger, Tempo, a vendor) without re-instrumenting.

# Minimal OpenTelemetry trace span in Python
from opentelemetry import trace

tracer = trace.get_tracer("checkout")

with tracer.start_as_current_span("charge_card") as span:
    span.set_attribute("provider", "stripe")
    span.set_attribute("amount_cents", 4999)
    charge = call_payment_provider()
    span.set_attribute("result", charge.status)

A common deployment is the OTel Collector: your apps send OTLP to a collector, which processes, batches, and routes to backends. It centralizes sampling and keeps backend choices out of your app code.

Sampling: you cannot store everything

At scale, tracing every request is expensive and mostly redundant. Sampling keeps a representative subset:

• Head-based sampling decides at the start of a request (e.g. keep 10%). Simple, but may miss rare errors.
• Tail-based sampling decides after the trace completes (e.g. keep all traces with errors or high latency). Smarter, more infrastructure.

For logs, the equivalents are sampling noisy info logs while keeping all errors, and setting retention tiers (hot for 7 days, cheap archive after).

SLOs: observability with a purpose

Telemetry without objectives is just noise. Service Level Objectives turn signals into decisions:

• Define an SLI (e.g. "% of requests served < 300ms").
• Set an SLO target (e.g. 99.5% over 30 days).
• Track the error budget (the 0.5% you're allowed to miss). When you burn it fast, alert; when you have budget, ship faster.

Alert on symptoms users feel (latency, errors, budget burn), not on every internal metric. Page-on-CPU is how teams get alert fatigue.

What this looks like on PandaStack

Observability is only useful if it's actually on. PandaStack captures it server-side so you don't bolt on an agent:

• Live build and app logs stream from a self-hosted Elasticsearch backend — you watch a deploy build in real time and tail your running app's logs.
• Metrics and analytics are captured server-side into ClickHouse with no client SDK to install. Performance metrics (latency, throughput) are kept distinct from audience analytics.

The deliberate choice: no SDK in your app for the platform-level signals. You still add OpenTelemetry inside your own code for deep tracing, but the baseline logs/metrics are there the moment you deploy.

A starter checklist

• [ ] Structured (JSON) logs with a trace_id field
• [ ] RED metrics per service, tracked at p50/p95/p99
• [ ] OpenTelemetry instrumentation + an OTel Collector
• [ ] Trace-ID propagation via traceparent
• [ ] Tail-based sampling that always keeps errors
• [ ] SLOs with error-budget alerting on user-facing symptoms

References

• [OpenTelemetry documentation](https://opentelemetry.io/docs/)
• [Google SRE Book — Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)
• [The RED Method (Grafana)](https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/)
• [W3C Trace Context specification](https://www.w3.org/TR/trace-context/)
• [Prometheus documentation](https://prometheus.io/docs/introduction/overview/)

---

Want live logs and server-side metrics the moment you deploy — no agent to install? PandaStack streams build/app logs and captures metrics out of the box. Try the free tier at [dashboard.pandastack.io](https://dashboard.pandastack.io).