How to Set Up Log Aggregation for Your Apps

Why aggregate logs at all

In a world of ephemeral containers and autoscaling, the logs you need are on a pod that was deleted ten minutes ago. SSHing into boxes to tail -f is dead. Log aggregation, collecting logs from every instance into a central, searchable store, is the only way to debug a distributed system. The goal: one place to search across all services, all instances, all time.

The log pipeline

Every log aggregation setup, no matter the tooling, follows the same four stages:

[App] --emit--> [Collector/Agent] --ship--> [Store/Index] --query--> [You]

1. 1Emit: your app writes structured logs to stdout/stderr.
2. 2Collect: an agent (Fluent Bit, Vector, the platform's collector) reads them.
3. 3Store + index: a backend (Elasticsearch/OpenSearch, Loki, ClickHouse) makes them searchable.
4. 4Query: you search, filter, and build dashboards.

Get stage 1 right and everything downstream gets easier.

Structured logging is non-negotiable

The single highest-leverage change you can make is to log JSON, not free text. Compare:

# Unstructured: hard to filter, easy to break parsing
2026-03-02 12:01 ERROR user login failed for bob from 1.2.3.4

{"ts":"2026-03-02T12:01:00Z","level":"error","msg":"login failed","user":"bob","ip":"1.2.3.4","request_id":"abc123"}

With structured logs you can query level:error AND user:bob, aggregate counts, and correlate by request_id. Most languages have a structured logger:

// Node with pino
const logger = require('pino')();
logger.error({ user: 'bob', ip: req.ip, request_id: req.id }, 'login failed');

# Python with structlog
import structlog
log = structlog.get_logger()
log.error('login failed', user='bob', ip=ip, request_id=rid)

Log to stdout, let the platform collect

Follow the twelve-factor rule: treat logs as event streams. Your app should write to stdout/stderr and never manage log files, rotation, or shipping itself. The runtime captures the stream; the collector ships it. This keeps your app stateless and portable.

// Don't do this in a container:
// fs.appendFile('/var/log/app.log', ...)  // ❌ stateful, lost on restart
// Just write to stdout:
console.log(JSON.stringify(entry));         // ✅ collector picks it up

The correlation ID pattern

The thing that makes aggregated logs *useful* rather than just centralized is a request/correlation ID. Generate one at the edge (or accept an inbound X-Request-ID), attach it to every log line for that request, and pass it to downstream services. Now you can trace a single user request across every service it touched with one query.

app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] || crypto.randomUUID();
  res.setHeader('x-request-id', req.id);
  next();
});

Choosing a backend

For most teams, the right answer is "whatever your platform already runs," because operating a logging backend at scale is a real job.

Controlling cost and noise

Log volume balloons fast, and storage and indexing cost money. Tactics:

• Set sane log levels. Don't ship debug from production by default.
• Sample high-volume, low-value logs (e.g., health-check hits).
• Set retention. Keep 7-30 days hot, archive or drop the rest.
• Don't log secrets or PII. It's a cost *and* a compliance liability.

Log aggregation on a managed platform

If you'd rather not run Fluent Bit and Elasticsearch yourself, a managed platform can do the collect-store-query loop for you. PandaStack ships live build and app logs backed by self-hosted Elasticsearch, so you see logs streaming from your containers in real time without configuring any agent. Because builds run in ephemeral Kubernetes Job pods and apps run as pods on GKE, the platform captures stdout/stderr for you and makes it searchable.

Your job on a managed platform is the part the platform can't do: emit structured JSON to stdout with a correlation ID. Do that, and centralized search becomes genuinely powerful instead of just a wall of text.

For metrics specifically (latency, error rates, throughput), PandaStack also captures server-side metrics and analytics into ClickHouse with no client SDK, so you don't have to instrument logging *and* metrics separately.

A minimal rollout plan

1. 1Switch your app to a structured (JSON) logger.
2. 2Add correlation-ID middleware and propagate the header downstream.
3. 3Write only to stdout/stderr; remove any file logging.
4. 4Set log levels per environment (debug in dev, info/warn in prod).
5. 5Confirm logs are searchable centrally and add a couple of saved queries for common incidents.
6. 6Set retention and check that you're not logging secrets.

Conclusion

Log aggregation isn't about a fancy tool, it's about discipline at the source: structured JSON, stdout-only, and a correlation ID threaded through every request. Get those three right and any backend, self-hosted or managed, becomes a debugging superpower. Skip them and you'll have a very expensive, very searchable pile of noise.

Want centralized, real-time logs without running the pipeline yourself? PandaStack streams live app logs out of the box, free tier included. Try it at https://dashboard.pandastack.io.

References

• The Twelve-Factor App, logs: https://12factor.net/logs
• Fluent Bit documentation: https://docs.fluentbit.io/manual
• Grafana Loki: https://grafana.com/docs/loki/latest/
• OpenSearch documentation: https://opensearch.org/docs/latest/
• OpenTelemetry logs: https://opentelemetry.io/docs/concepts/signals/logs/