What Is Elasticsearch for Log Aggregation?

When one app runs across many ephemeral pods, "just SSH in and grep the log file" stops working — the pod may already be gone. Log aggregation solves this by shipping every log line to a central store you can search. Elasticsearch is one of the most popular engines for that store. This guide explains what it does and why it's good at logs.

The problem: logs are scattered and ephemeral

In a modern platform:

• Apps run in many replicas across many nodes.
• Containers are ephemeral — a pod that crashed took its local logs with it.
• Builds run in throwaway pods that disappear when the build ends.

You can't debug by grepping files on a server, because there's no single server and the files don't persist. You need logs *collected centrally* while the workload is still alive.

What log aggregation is

Log aggregation is a pipeline:

app/build stdout → collector/shipper → central store (index) → search UI

Each component writes to stdout/stderr; a shipper tails that output and forwards it; a store indexes it; and you query the store. The classic stack is ELK — Elasticsearch (store/search), Logstash or a lighter shipper (collect/transform), and Kibana (visualize) — though many shippers (Filebeat, Fluent Bit, Vector) can replace Logstash.

What Elasticsearch is

Elasticsearch is a distributed, document-oriented search and analytics engine built on Apache Lucene. You send it JSON documents (a log line becomes a document with fields like timestamp, level, message, service), and it indexes them for fast retrieval and full-text search.

The magic is the inverted index.

Why search is fast: the inverted index

A naive search scans every line for your term — O(n) over all your logs, which is hopeless at scale. Elasticsearch instead builds an *inverted index*: a map from each term to the list of documents containing it.

"error"   → [doc 3, doc 17, doc 942, ...]
"timeout" → [doc 17, doc 88, ...]
"137"     → [doc 942, ...]

Searching for error timeout becomes a fast intersection of two precomputed lists rather than a scan of the whole corpus. This is the same idea that makes web search fast, applied to your logs. It's why you can search millions of log lines and get results in milliseconds.

Key concepts

Structured logging makes this shine

Elasticsearch is good at full-text search, but it's *great* when your logs are structured JSON. Compare:

# Unstructured
User 42 failed login from 10.0.0.5

# Structured
{"event":"login_failed","user_id":42,"ip":"10.0.0.5","level":"warn"}

With structured logs you can query by field — user_id:42 AND event:login_failed — and build aggregations (counts, histograms) instead of relying on fragile substring matching. Emitting JSON logs from your app pays off enormously here.

Time-based indices and lifecycle

Logs are append-only and time-ordered, so Elasticsearch deployments usually create one index per time window (per day, say). This makes lifecycle management clean:

• Hot recent indices stay fast and queryable.
• Old indices get archived or deleted via Index Lifecycle Management (ILM).

This is how you keep storage bounded — you don't keep every log forever; you set a retention window appropriate to your needs.

How PandaStack uses Elasticsearch

PandaStack runs a self-hosted Elasticsearch cluster as the backing store for logs. Two streams flow into it:

• Build logs from ephemeral build pods — shipped as the build runs, so they survive the pod's destruction and remain searchable afterward.
• App (runtime) logs from your running containers.

This is what makes the dashboard's logs both live (streamed as written) and searchable (indexed for fast term lookup) — you can tail a build in real time and later search across past deploys for a specific error string. (See our live build logs tutorial for the practical search workflow.)

Operational considerations

If you run Elasticsearch yourself, budget for:

• Disk: logs are voluminous; size storage and set retention.
• Memory: Lucene loves RAM; the JVM heap and OS page cache both matter.
• Shard hygiene: too many tiny shards hurts performance; right-size them.
• Mappings: avoid mapping explosions from highly variable JSON fields.

These are real costs — which is part of why a managed platform absorbing this for you is valuable: you get search without running the cluster.

References

• [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)
• [Apache Lucene (inverted index)](https://lucene.apache.org/core/)
• [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)
• [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)
• [Fluent Bit log shipper](https://docs.fluentbit.io/manual)

---

PandaStack's self-hosted Elasticsearch gives you live, searchable build and app logs without operating a cluster yourself. Push a repo and search your first build at [dashboard.pandastack.io](https://dashboard.pandastack.io).