How to Monitor Application Performance Metrics

Start with the right metrics

Most teams drown in dashboards full of metrics nobody acts on. The fix is to start from proven frameworks rather than graphing everything you can.

The RED method (for request-driven services)

For any service that handles requests, track three things:

• Rate — requests per second.
• Errors — failed requests per second (and error rate as a percentage).
• Duration — latency distribution.

The USE method (for resources)

For each resource (CPU, memory, disk, network):

• Utilization — % busy.
• Saturation — how much work is queued/waiting.
• Errors — error counts.

Between RED (what users experience) and USE (what your infrastructure is doing), you cover the vast majority of "why is it slow?" investigations.

Percentiles, not averages

The single most important measurement habit: stop looking at average latency. Averages hide the pain. If 95% of requests are fast and 5% take 10 seconds, your average looks fine while a meaningful slice of users suffers.

Track percentiles instead:

A healthy service has p50 and p99 reasonably close. A large gap means inconsistent performance, usually from contention, garbage collection, cold caches, or a slow dependency.

The four golden signals

Google's SRE book distills monitoring to four signals worth memorizing: latency, traffic, errors, and saturation. If you only build four graphs, build those. They map cleanly onto RED + saturation from USE.

Instrumenting your app

The traditional path is a metrics library that exposes a /metrics endpoint scraped by Prometheus:

// Node with prom-client
const client = require('prom-client');
const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Request duration',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.01, 0.05, 0.1, 0.3, 1, 3, 10],
});

app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on('finish', () => end({ method: req.method, route: req.route?.path, status: res.statusCode }));
  next();
});

Histogram buckets are what let you compute percentiles later, choose buckets that bracket your expected latencies.

Metrics vs. logs vs. traces

These three pillars answer different questions:

• Metrics: aggregate numbers over time ("p99 latency is up"). Cheap, great for alerting and dashboards.
• Logs: discrete events ("this request failed because X"). Great for root cause.
• Traces: a single request's path across services. Great for "where is the time going?"

Good observability uses all three: metrics tell you *something* is wrong, traces tell you *where*, logs tell you *why*.

Monitoring without hand-instrumenting everything

The traditional approach requires adding an SDK to every service, exposing /metrics, running Prometheus, and building Grafana dashboards. That's powerful but it's real work, and it's easy to forget to instrument a new service.

An alternative is server-side capture at the ingress layer. Because all traffic flows through an ingress/proxy, you can measure rate, errors, and latency for every service without touching application code. PandaStack does exactly this: it captures server-side metrics and analytics into ClickHouse with no client SDK, using Kong ingress and a proxy. You get request rates, status codes, and latency for your deployed apps automatically, and ClickHouse is fast enough to compute percentiles over large volumes.

This doesn't replace deep, custom application metrics (you'll still want a histogram around that one critical business operation), but it means every service is observable the moment it's deployed, not after someone remembers to add instrumentation.

Setting alerts that don't cry wolf

Alert on symptoms users feel, not causes:

• Good: "p99 latency > 2s for 5 minutes" or "error rate > 2% for 5 minutes."
• Noisy: "CPU > 80%" (often fine and self-correcting).

Use multi-window or sustained-duration conditions so a single spike doesn't page you. Tie alert thresholds to your SLOs, the error budget tells you when to actually wake someone.

A practical dashboard layout

1. 1Top row, golden signals: request rate, error rate, p50/p95/p99 latency, saturation.
2. 2Per-endpoint breakdown: which routes are slow or erroring.
3. 3Dependencies: database query time, cache hit rate, downstream API latency.
4. 4Resources: CPU, memory, restarts per service.

Conclusion

Effective performance monitoring is less about tooling and more about measuring the right things: golden signals, percentiles over averages, and alerts tied to user-felt symptoms. Whether you instrument with Prometheus or lean on server-side capture, the discipline is the same, watch what users experience, and make every new service observable from day one.

PandaStack captures server-side metrics into ClickHouse automatically for every deployed app, no SDK required. See it on your own service with the free tier at https://dashboard.pandastack.io.

References

• Google SRE book, monitoring distributed systems: https://sre.google/sre-book/monitoring-distributed-systems/
• The RED method: https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/
• The USE method (Brendan Gregg): https://www.brendangregg.com/usemethod.html
• Prometheus documentation: https://prometheus.io/docs/introduction/overview/
• OpenTelemetry metrics: https://opentelemetry.io/docs/concepts/signals/metrics/