ELK Stack Explained: Internals, Pipelines and Production Failures

Every production system lies. Not intentionally — but without proper observability, your application fails silently, degrades mysteriously, and wakes you at 3am with zero context. Log files exist, but a 400GB flat log file on a server nobody SSHs into anymore is just expensive noise. The ELK Stack transforms that noise into signal: structured, searchable, visualized intelligence about everything your infrastructure is doing, in real time.

The core problem ELK solves is the gap between raw log data and actionable insight. A typical microservices platform produces logs from dozens of services, each in a slightly different format, scattered across hundreds of containers. Correlating a failed payment transaction across an API gateway, an auth service, a Kafka consumer, and a Postgres adapter — without a centralized log aggregation system — is an exercise in madness. ELK gives every log line a home, a shape, and a timeline.

By the end you will understand how Elasticsearch actually indexes and retrieves documents under the hood, how to build Logstash pipelines that handle real-world log formats including multiline stacktraces, how to design Kibana dashboards that answer operational questions rather than just looking impressive in a quarterly review, and exactly where production deployments fall apart and how to prevent it. The incidents in this article are real. The fixes are the ones that actually worked.

What ELK Stack Is and How the Components Connect

ELK is not three tools bolted together. It is a data pipeline with three distinct failure domains, and understanding how data flows between them is what separates engineers who can debug it from engineers who restart services and hope.

Data originates on your hosts and containers. Filebeat — a lightweight Go agent — tails log files and ships events forward. It is stateful: Filebeat maintains a registry file tracking its read position in every file it monitors. If that registry file is corrupted by an unclean shutdown (common on spot instances), Filebeat loses its position and either re-ships everything from the start or skips forward to the current file end, depending on configuration. Always run Filebeat with its registry on a persistent volume and set close_inactive to a sensible value so file handles do not accumulate.

Filebeat ships to Logstash, or — in high-volume environments — to Kafka first. The Kafka buffer is not optional at scale. It absorbs traffic spikes so Logstash does not receive a 10x burst and OOM. It also means a Logstash restart does not lose data — Kafka holds the events until Logstash recovers and resumes consuming from its committed offset. Running Logstash reading directly from files in a high-volume environment is fragile. Add Kafka as the buffer between collection and processing.

Logstash reads from Kafka, applies filters to parse and enrich each event, and writes structured documents to Elasticsearch. The pipeline is: inputs -> filters -> outputs. Each stage runs in its own thread pool. The filter stage is where CPU is spent and where most production problems originate.

Elasticsearch receives structured JSON documents, indexes them into an inverted index, and serves search and aggregation queries. Kibana connects to Elasticsearch and renders the results.

The triage order when logs stop flowing is always: Elasticsearch first, then Logstash, then Kibana. Storage failures cascade upstream. A healthy Logstash shipping to a broken Elasticsearch looks, from the outside, identical to a broken Logstash — events simply stop appearing in Kibana. Check ES health before anything else.

In 2026, Elastic also offers OpenTelemetry-native ingestion and the Elastic Agent as a replacement for the Filebeat plus Logstash combination. The Elastic Agent consolidates collection and processing into a single managed binary with central policy control through Fleet. For new deployments, evaluate Elastic Agent rather than defaulting to the classic Filebeat-Logstash split. For existing deployments, migration is straightforward but not mandatory — the classic stack still works and is fully supported.

# Minimal ELK stack for local development and testing
# Shows the actual data flow: Filebeat -> Logstash -> Elasticsearch <- Kibana
# For production, replace single-node ES with a proper cluster and add Kafka between Filebeat and Logstash

version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false       # Dev only — always enable security in production
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
      - cluster.routing.allocation.disk.watermark.low=85%
      - cluster.routing.allocation.disk.watermark.high=90%
      - cluster.routing.allocation.disk.watermark.flood_stage=95%  # Read-only at 95%
    ports:
      - "9200:9200"
    volumes:
      - esdata:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD-SHELL", "curl -sf 'localhost:9200/_cluster/health' | grep -v '\"status\":\"red\"'"]
      interval: 10s
      timeout: 5s
      retries: 10

  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
    environment:
      - LS_JAVA_OPTS=-Xms1g -Xmx2g
    ports:
      - "5044:5044"  # Beats input
    depends_on:
      elasticsearch:
        condition: service_healthy

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.13.0
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - filebeatdata:/usr/share/filebeat/data  # Persistent registry — survives container restarts
    depends_on:
      - logstash

volumes:
  esdata:
  filebeatdata:  # Registry persistence prevents re-shipping on restart

Beats Family — Filebeat, Metricbeat, Packetbeat, and When to Use Each

The Beats family is Elastic's collection of lightweight data shippers. Each Beat is purpose-built for a specific data type — logs, metrics, network packets — and runs as a single binary with minimal configuration. Understanding which Beat to use for which job prevents the mistake of forcing Filebeat to collect metrics or Metricbeat to tail log files.

Filebeat is the workhorse for log collection. It tails files, follows symlinks, handles rotation, and ships raw log lines to Logstash or directly to Elasticsearch. It maintains a registry — a local file tracking read positions — so a restart does not re-ship the same lines. Filebeat supports multiline aggregation, which is critical for Java stacktraces. Configure multiline in Filebeat rather than Logstash whenever possible to reduce Logstash heap pressure.

Metricbeat collects system and service metrics. It runs modules that know how to talk to specific services — MySQL, PostgreSQL, Redis, Nginx, Kafka, Docker, Kubernetes. Metricbeat pulls metrics from each module on a configurable period. The output is numerical time-series data, not raw log lines. Do not use Filebeat to read /proc/stats — use Metricbeat with the system module.

Packetbeat captures and parses network traffic. It runs as a packet sniffer using libpcap (Linux) or WinPcap (Windows), decoding protocols like HTTP, MySQL, PostgreSQL, Redis, Thrift, and DNS. Packetbeat reconstructs full transactions from packets, so it can show you every SQL query or HTTP request/response pair that crosses your network segment.

Auditbeat collects security audit events from your Linux kernel using the Linux Audit Framework. It ships user logins, privilege escalations (sudo), file integrity events (when critical configs change), and process execution logs. Auditbeat is the right tool for compliance auditing (SOC2, PCI-DSS) and security monitoring.

Heartbeat performs uptime monitoring. It pings services (ICMP), connects to TCP ports, or checks HTTP endpoints for expected status codes and response body patterns. Heartbeat sends synthetic check results as documents, which you can alert on for service availability. It is not a log collector and has no relation to a human heartbeat — it is named for the regular 'heartbeat' signal it emits.

Winlogbeat captures Windows Event Logs — Application, Security, Setup, System, and forwarded events. If your infrastructure includes Windows servers, Winlogbeat is the only supported way to get Windows Event Logs into Elasticsearch reliably. Do not try to tail raw .evtx files with Filebeat.

# beats-comparison.yml — Quick reference for choosing the right Beat

# ============================================================
# FILEBEAT — Log files (application logs, JSON logs, plaintext)
# ============================================================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/app/*.log
  multiline:
    type: pattern
    pattern: '^\d{4}-\d{2}-\d{2}'  # New log lines start with timestamp
    negate: true
    match: after

# ============================================================
# METRICBEAT — System metrics + service metrics
# ============================================================
metricbeat.modules:
- module: system
  period: 10s
  metricsets: ["cpu", "memory", "diskio", "filesystem", "load", "process"]
- module: nginx
  period: 30s
  hosts: ["http://localhost/status"]
- module: docker
  period: 10s
  hosts: ["unix:///var/run/docker.sock"]

# ============================================================
# PACKETBEAT — Network protocol analysis
# ============================================================
packetbeat.interfaces.device: eth0
packetbeat.protocols:
- type: http
  ports: [80, 8080]
  send_headers: ["Authorization"]
- type: mysql
  ports: [3306]
- type: pgsql
  ports: [5432]
- type: redis
  ports: [6379]

# ============================================================
# WINLOGBAT — Windows Event Logs
# ============================================================
winlogbeat.event_logs:
- name: Application
  ignore_older: 72h
- name: Security
  ignore_older: 72h
- name: System
- name: Setup

# ============================================================
# HEARTBEAT — Uptime monitoring
# ============================================================
heartbeat.monitors:
- type: http
  name: Production API
  urls: ["https://api.example.com/health"]
  schedule: '@every 30s'
  check.response.status: [200]
- type: tcp
  name: MySQL
  hosts: ["db.example.com:3306"]
  schedule: '@every 1m'

# ============================================================
# AUDITBEAT — Linux audit framework
# ============================================================
auditbeat.modules:
- module: auditd
  audit_rules: |
    -w /etc/passwd -p wa -k identity
    -w /etc/nginx -p wa -k config
    -a always,exit -S execve -k process_execution

How Elasticsearch Actually Indexes Documents — Inverted Indices Under the Hood

Elasticsearch does not search documents. It searches an inverted index — a data structure that maps every unique term to the list of documents that contain it. When you index a document, Elasticsearch tokenizes the text, normalizes case, applies stemming if configured, and writes each token into a term dictionary. The term dictionary points to a postings list: document IDs, term frequency, and position offsets.

This is why Elasticsearch is fast at full-text search. You are not scanning every document. You are looking up a term in a sorted dictionary and getting back a pre-computed list of matching document IDs. BM25 scoring then ranks those matches by term frequency, inverse document frequency, and field length normalization.

Each Elasticsearch shard is an independent Lucene index. Lucene segments are immutable — once written, they never change. New or updated documents go into an in-memory buffer, then get flushed to a new segment on refresh, which defaults to every 1 second. This means there is a 1-second window where a newly indexed document is not yet searchable. If you need sub-second search freshness, the answer is not lowering the refresh interval — it will kill indexing throughput because every refresh triggers segment creation and eventual merges.

Segment merging happens in the background and is a silent performance killer when misconfigured. Too many small segments accumulate when indexing is faster than merging. The merge thread then consumes I/O and CPU, spiking latency for active searches. Monitor segment count per shard with _cat/segments — more than 100 segments per shard is a sign your merge policy needs tuning. For bulk indexing jobs, set refresh_interval to 30s or -1 during the load, force a refresh when done, then restore the interval.

Field mapping is where most teams create invisible performance problems. Every field you add increases the inverted index size and slows indexing. Use dynamic: strict in your index templates to reject unexpected fields and define only the fields you actually search or aggregate on. A common mistake is indexing full HTTP request bodies as a single text field, then wondering why searches are slow. Use index: false for fields you store but never query.

High-cardinality keyword fields deserve specific mention. Trace IDs, request IDs, and session tokens as keyword fields create term dictionaries with millions of unique values that cannot fit in RAM. Every search touching those fields forces disk lookups. Either do not index them as keywords, or use a separate index with appropriate settings for correlation lookups rather than mixing them into your main search index.

#!/bin/bash
# See exactly how Elasticsearch tokenizes and stores a document
# The _termvectors API exposes the inverted index directly

# Index a sample document
curl -s -XPOST 'localhost:9200/logs/_doc/1' \
  -H 'Content-Type: application/json' \
  -d '{
    "message": "Payment failed for user 4821 timeout after 30s",
    "service": "payment-api",
    "level": "error",
    "@timestamp": "2026-04-25T10:30:00Z"
  }'

# Force refresh so the document is searchable immediately
curl -s -XPOST 'localhost:9200/logs/_refresh'

# See how ES tokenized the message field — this IS the inverted index
curl -s -XGET 'localhost:9200/logs/_termvectors/1?fields=message&pretty'
# Output shows each token, its frequency, and its position:
# "payment"  -> term_freq: 1, position: 0
# "failed"   -> term_freq: 1, position: 1
# "user"     -> term_freq: 1, position: 3
# "timeout"  -> term_freq: 1, position: 5

# Search uses the inverted index — no full document scan
curl -s -XGET 'localhost:9200/logs/_search?pretty' \
  -H 'Content-Type: application/json' \
  -d '{"query": {"match": {"message": "timeout"}}}'

Logstash Pipelines — Ingest, Transform, Ship and Where They Break

Logstash receives raw data from inputs, transforms it through filters, and ships structured events to outputs. The pipeline is linear — input -> filter -> output — with each stage running in its own thread pool. The number of worker threads processing the filter stage is controlled by pipeline.workers, which defaults to the number of available CPU cores. Understanding the threading model is the first step to understanding why pipelines stall.

The Grok filter is where most Logstash performance problems originate. Grok combines regular expressions with named capture groups to extract structured fields from unstructured text. A pattern like %{COMBINEDAPACHELOG} expands to a 200-character-plus regex. When your log format does not match the pattern, Grok tries every alternative before failing. In a pipeline processing 10,000 events per second with a 5% failure rate, that is 500 wasted regex evaluations per second. Always add a catch-all pattern as the last alternative: %{GREEDYDATA:log_message}. It ensures events flow through even on mismatch, and you tag the failure for visibility rather than silently dropping the event.

Multiline event handling is the second major trap. Java stacktraces and Python tracebacks span multiple lines. Logstash's multiline codec aggregates them into a single event by buffering pending lines in JVM heap. A burst of stacktraces from a crashing service — which is exactly when you most need your logs — can spike heap usage from 500MB to 3GB in under a minute. Set -Xmx to at least 4GB when using multiline in production. Reduce max_lines to a realistic ceiling (200 is usually enough for stacktraces) so a runaway exception chain cannot consume unlimited heap.

Dead letter queues are the safety net that most teams skip and regret. By default, Logstash silently drops documents that Elasticsearch rejects — mapping conflicts, disk blocks, field limit breaches. Enable it in logstash.yml: dead_letter_queue.enable: true and dead_letter_queue.max_bytes: 1024mb. The DLQ is a local directory on the Logstash host, not an Elasticsearch index. Inspect it at the path configured by path.dead_letter_queue (default: /var/lib/logstash/dead_letter_queue). Use the dead_letter_queue input plugin to replay rejected events after fixing the root cause.

The pipeline.ordered setting deserves explicit mention. By default it is set to auto, which enables ordered processing when pipeline.workers is 1 and disables it otherwise. Set pipeline.ordered: false explicitly when event ordering between inputs does not matter — it allows workers to process events without coordination overhead, improving throughput at the cost of delivery order guarantees. For log pipelines where Elasticsearch timestamps handle ordering at query time, this is almost always the right call.

Pipeline workers and batch size interact directly with throughput. For a 16-core machine, start with 8 workers and a batch size of 250. Increasing batch size improves throughput by amortizing per-batch overhead but increases per-event latency and heap usage. A batch that fills with slow-to-process multiline events holds the worker thread for longer, starving other events. Benchmark with realistic load before committing to any setting.

# /etc/logstash/conf.d/production-logs.conf
# Production Logstash pipeline for Java microservices
# Handles both structured JSON logs and raw stacktraces with multiline

input {
  # Beats input for Filebeat agents on each host
  beats {
    port => 5044
    congestion_threshold => 5  # Backpressure when 5 Filebeat connections are queued
  }
}

filter {
  # Attempt JSON parse first — structured logs from Logback/Jackson need no Grok
  json {
    source  => "message"
    target  => "parsed"
    skip_on_invalid_json => true  # Keep raw message if not JSON — do not drop it
  }

  # Grok fallback for non-JSON logs and raw stacktraces
  if ![parsed] {
    grok {
      match => {
        "message" => [
          # Primary pattern: structured log line with thread and logger
          "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:thread}\] %{DATA:logger} - %{GREEDYDATA:log_message}",
          # Catch-all: never drop an event — tag it and keep flowing
          # Without this, every unmatched line is silently discarded
          "%{GREEDYDATA:log_message}"
        ]
      }
      tag_on_failure => ["_grokparsefailure_custom"]
    }
  }

  # Multiline handling for Java stacktraces
  # max_bytes limits heap consumption from runaway exception chains
  # This is a codec-level setting; shown here as a reference for input configuration
  # In practice, configure multiline on the Filebeat side to reduce Logstash heap pressure

  # Normalize timestamp to ISO 8601 for Elasticsearch
  date {
    match         => ["timestamp", "ISO8601", "yyyy-MM-dd HH:mm:ss.SSS"]
    target        => "@timestamp"
    remove_field  => ["timestamp"]
  }

  # Add processing metadata so you can trace which pipeline version and instance handled an event
  mutate {
    add_field => {
      "pipeline_version" => "4.0"
      "processed_by"    => "%{[host][name]}"
    }
  }
}

output {
  elasticsearch {
    hosts                  => ["es-data-01:9200", "es-data-02:9200", "es-data-03:9200"]
    index                  => "app-logs-%{+YYYY.MM.dd}"
    # Retry settings — transient ES errors should not cause data loss
    retry_max_interval     => 30
    retry_initial_interval => 2
  }

  # Dead letter queue is configured in logstash.yml, not here.
  # Add to logstash.yml:
  #   dead_letter_queue.enable: true
  #   dead_letter_queue.max_bytes: 1024mb
  #   path.dead_letter_queue: /var/lib/logstash/dead_letter_queue
  # Inspect the DLQ directory to see what ES rejected.
  # Replay with the dead_letter_queue input plugin after fixing the root cause.
}

Logstash Filter Cheat Sheet — Grok, Date, Mutate, GeoIP

Logstash filters transform raw events into structured documents before they reach Elasticsearch. The four most frequently used filters in production pipelines are Grok, Date, Mutate, and GeoIP. Having a scannable reference makes pipeline debugging faster and reduces the guesswork when logs show up with missing fields or wrong timestamps.

Grok extracts structured fields from unstructured text using pattern matching. Built-in patterns cover common formats — %{COMBINEDAPACHELOG}, %{TIMESTAMP_ISO8601}, %{LOGLEVEL}. For custom formats, compose smaller patterns. Always add a catch-all as the last alternative to prevent dropped events.

Date parses timestamp strings from your logs into the @timestamp field. If you skip this, Elasticsearch uses the current time at indexing, making log order unreliable. The match parameter takes an array of format strings to try in order.

Mutate modifies field values and structures — renaming, copying, converting types, removing fields, and adding static strings. Use it to normalize field names across services (e.g., renaming customer_email to email) or to add pipeline metadata.

GeoIP enriches events with geographical location data from an IP address. It adds fields like geoip.country_code2, geoip.city_name, and geoip.location. This only works with public IPs. Rate-limit usage because the GeoIP database update can become a performance overhead on high-volume pipelines.

Grok failure debugging is where most teams waste time. When your pattern does not match, Logstash adds a _grokparsefailure tag to the event. Check for these tags in Kibana. Use the Grok Debugger in Kibana Dev Tools to test patterns against actual log lines before deploying.

# /etc/logstash/conf.d/filters.conf
# Common Logstash filter patterns — copy, paste, modify

# ============================================================
# 1. GROK — Extract structured fields from unstructured text
# ============================================================
filter {
  grok {
    match => {
      "message" => [
        # Apache/Nginx combined log format
        "%{COMBINEDAPACHELOG}",
        # Custom JSON-like log line
        "timestamp=%{TIMESTAMP_ISO8601:timestamp} level=%{LOGLEVEL:level} trace_id=%{UUID:trace_id} msg=%{GREEDYDATA:message}",
        # Catch-all: never drop events — tag them instead
        "%{GREEDYDATA:log_message}"
      ]
    }
    # Tag failures so you can see them in Kibana
    tag_on_failure => ["_grokparsefailure_custom"]
    # Remove the original message after extracting to save disk
    remove_field => ["message"]
  }
}

# ============================================================
# 2. DATE — Parse timestamp into @timestamp
# ============================================================
filter {
  date {
    # Try these formats in order until one matches
    match => [
      "timestamp",
      "ISO8601",
      "yyyy-MM-dd HH:mm:ss.SSS",
      "dd/MMM/yyyy:HH:mm:ss Z"  # Apache log format
    ]
    target => "@timestamp"
    # Remove the original timestamp field after parsing
    remove_field => ["timestamp"]
  }
}

# ============================================================
# 3. MUTATE — Modify, rename, convert, or add fields
# ============================================================
filter {
  mutate {
    # Rename a field to standardize across services
    rename => {
      "customerEmail" => "email"
      "source_host"   => "hostname"
    }
    # Convert string numbers to actual numeric types
    convert => {
      "status_code" => "integer"
      "duration_ms" => "float"
    }
    # Remove fields you never query
    remove_field => ["headers", "raw_body"]
    # Add static processing metadata
    add_field => {
      "pipeline_name" => "logs-prod"
      "environment"   => "production"
    }
    # Copy a value to a new field
    copy => {
      "user_id" => "user.id"
    }
  }
}

# ============================================================
# 4. GEOIP — Enrich with location data from IP address
# ============================================================
filter {
  geoip {
    # Source field containing the IP address
    source => "client_ip"
    # Fields to add (default includes country, city, location, etc.)
    target => "geoip"
    # Skip if IP is private (10.x.x.x, 192.168.x.x, 172.16.x.x)
    # Not a filter setting — handle with a conditional before this filter
  }
}

# ============================================================
# 5. IF — Conditional pipelines
# ============================================================
filter {
  # Only apply heavy filters to error logs (10% of traffic)
  if [level] == "ERROR" or [status_code] >= 500 {
    grok {
      match => { "stacktrace" => "%{JAVASTACKTRACE}" }
    }
  }
  
  # Skip GeoIP for private IP addresses
  if [client_ip] !~ /^(10\.|172\.1[6-9]|172\.2[0-9]|172\.3[0-1]|192\.168\.)/ {
    geoip {
      source => "client_ip"
    }
  }
}

# ============================================================
# 6. KV (Key-Value) — Parse key=value pairs
# ============================================================
filter {
  kv {
    # Source field containing key=value pairs
    source => "query_string"
    # Target field for extracted fields
    target => "params"
    # Separator between key and value (default '=')
    field_split => "&"
    value_split => "="
    # Remove the source after extraction
    remove_field => ["query_string"]
  }
}

Kibana Dashboards That Actually Answer Questions

Most Kibana dashboards are digital art. They look impressive in a demo and answer nothing during an incident. A dashboard with 47 panels showing every possible metric is a distraction when you are trying to figure out why payments are failing at 2am.

The right approach is to start with the question, then build the visualization. 'Which services are returning 5xx errors in the last 15 minutes?' needs one metric — error count — one dimension — service name — and one filter — status code 500 or above, time range last 15 minutes. That is a single data table, not a 12-panel dashboard. The panel answers the question. Everything else is friction.

Kibana's index patterns are the second major gotcha. When you create an index pattern like app-logs-*, Kibana fetches field mappings from every matching index. If you have 90 days of daily indices with dynamic mapping enabled, you can easily have 3,000-plus fields — especially when different services log different JSON structures that Elasticsearch ingests as separate mapped fields. Every time someone opens Discover or creates a visualization, Kibana loads all those field definitions. That is why your dashboard takes 20 seconds to load. The fix is upstream: use dynamic: strict in your index template and define only the fields you actually query.

For real incident response, saved searches outperform visualizations. They load faster because they do not aggregate — they list raw log lines with column selections. Pin a few key saved searches at the top of your Kibana navigation: 5xx errors, slow queries over 5 seconds, auth failures. When something breaks, open the relevant saved search rather than waiting for a complex dashboard to render aggregations across 90 days of data.

Kibana Query Language over Lucene syntax is worth the initial learning curve. KQL is more readable, less error-prone when written quickly under pressure, and better supported in autocomplete. Train the whole team on a handful of patterns — field:value, field:* wildcards, AND/OR combinations, range queries with > and < — and you will have faster incident investigation.

Time series visualizations with a large time range are a silent performance killer. Querying 30 days of data with a 1-minute bucket interval generates 43,200 buckets. Elasticsearch computes all of them. Use auto-interval on date histograms for overview dashboards and a fixed short interval only when drilling into a specific incident window. The inspect button on any Kibana panel shows the raw Elasticsearch query being executed — this is invaluable for understanding why a dashboard is slow or showing unexpected data.

// Kibana saved search for incident triage — 5xx errors by service
// Import via Kibana -> Stack Management -> Saved Objects -> Import
// This is the first thing to open when someone says 'something is broken'
{
  "attributes": {
    "title": "5xx Errors by Service — Last 15m",
    "description": "Real-time error view for incident triage. Open this first, not the overview dashboard.",
    "columns": ["@timestamp", "service", "level", "log_message", "status_code"],
    "sort": [["@timestamp", "desc"]],
    "kibanaSavedObjectMeta": {
      "searchSourceJSON": "{\"query\":{\"query\":\"status_code >= 500\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"index\":\"app-logs-*\",\"type\":\"range\"},\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-15m\"}}}}]}"
    }
  },
  "type": "search"
}

// Human-readable version of the embedded query above:
// {
//   query: { query: 'status_code >= 500', language: 'kuery' },
//   filter: [{ range: { '@timestamp': { gte: 'now-15m' } } }]
// }
//
// The nested JSON escaping in kibanaSavedObjectMeta is the Kibana saved objects
// wire format — this is correct and required for import. The readable version above
// shows what is actually being executed against Elasticsearch.

Kibana Query Language (KQL) vs Lucene — Syntax Comparison

Kibana gives you two query language options: Kibana Query Language (KQL) and Lucene. KQL is the default in modern Kibana versions (7.0+) and is the recommended choice for most users. Lucene is the legacy syntax that powers Elasticsearch's underlying query parser. Knowing both is useful for debugging, but KQL should be your daily driver.

KQL is designed for discoverability and error resistance. It provides autocomplete suggestions, syntax highlighting, and immediate error feedback. You cannot write invalid KQL — Kibana tells you where the syntax breaks. KQL supports nested fields, existence checks, and range queries with a cleaner syntax. Use KQL for all ad-hoc exploration and dashboard panels.

Lucene is more powerful but more dangerous. It supports regex, fuzzy queries, and proximity searches that KQL does not. The trade-off is that Lucene does not prevent you from writing queries that are syntactically valid but semantically wrong. A misplaced parenthesis can change the entire query meaning without an error message. Reserve Lucene for advanced use cases where KQL falls short, and always test Lucene queries in Dev Tools before adding them to dashboards.

The field:value syntax is identical in both languages. Wildcards work in both — service:payment* matches payment-api, payment-processor, payment-service. The differences appear in ranges, existence checks, and complex Boolean logic.

# KQL vs Lucene — Syntax Comparison Quick Reference
# Use KQL by default. Only switch to Lucene when you need advanced features.

# ============================================================
# BASIC FIELD LOOKUP — Same in both
# ============================================================
# Match exact field value
KQL:     status_code: 500
Lucene:  status_code:500

# Match value anywhere in full-text field (message analysis applies)
KQL:     message: "timeout"
Lucene:  message:timeout

# ============================================================
# WILDCARDS — Same in both
# ============================================================
# Prefix match
KQL:     service: payment*
Lucene:  service:payment*

# Single character wildcard
KQL:     trace_id: 1a2b?ef*
Lucene:  trace_id:1a2b?ef*

# ============================================================
# RANGE QUERIES — KQL is more readable
# ============================================================
# Greater than or equal
KQL:     duration_ms >= 5000
Lucene:  duration_ms:[5000 TO *]

# Between (inclusive both ends)
KQL:     status_code >= 400 and status_code <= 499
Lucene:  status_code:[400 TO 499]

# Between (exclusive upper bound)
KQL:     duration_ms >= 0 and duration_ms < 1000
Lucene:  duration_ms:[0 TO 1000}

# Date range
KQL:     @timestamp >= "2026-04-25T10:00:00"
Lucene:  @timestamp:[2026-04-25T10:00:00 TO *]

# ============================================================
# BOOLEAN LOGIC — KQL uses words, Lucene uses symbols
# ============================================================
# AND
KQL:     level: ERROR AND service: payment-api
Lucene:  level:ERROR AND service:payment-api

# OR (note: KQL requires uppercase OR)
KQL:     level: ERROR OR level: WARN
Lucene:  level:ERROR OR level:WARN

# NOT
KQL:     NOT level: DEBUG
Lucene:  -level:DEBUG OR NOT level:DEBUG

# Complex grouping
KQL:     (status_code >= 500 OR level: ERROR) AND service: payment*
Lucene:  (status_code:[500 TO *] OR level:ERROR) AND service:payment*

# ============================================================
# EXISTENCE CHECKS — KQL is more readable
# ============================================================
# Field exists (has any value, including null)
KQL:     trace_id: *
Lucene:  _exists_:trace_id

# Field does NOT exist
KQL:     NOT trace_id: *
Lucene:  -_exists_:trace_id

# ============================================================
# NESTED FIELDS — Same in both (dot notation)
# ============================================================
KQL:     geoip.country_code: US
Lucene:  geoip.country_code:US

# ============================================================
# LUCENE-ONLY FEATURES (not available in KQL)
# ============================================================
# Regular expressions (use sparingly — expensive)
Lucene:  message:/pay.ent.*/i

# Fuzzy queries (character edit distance)
Lucene:  customer_name:bob~1

# Proximity searches
Lucene:  "user created"~3

# Boosting terms
Lucene:  level:ERROR^2 OR level:WARN

# ============================================================
# REAL-WORLD INCIDENT QUERIES
# ============================================================
# Find all errors from payment service in last hour (KQL)
level: ERROR AND service: payment-* AND @timestamp >= now-1h

# Find slow API calls (>10s) excluding health checks (Lucene)
duration_ms:[10000 TO *] AND NOT endpoint:/health

# Find any 5xx or ERROR from payment services except test users (KQL)
(status_code >= 500 OR level: ERROR) AND service: payment-* AND NOT user_id: test-*

Shard Strategy and Capacity Planning — The Decisions That Haunt You Later

Shard count is the most consequential decision in an Elasticsearch deployment and it is almost always wrong on the first try. Too many shards and your cluster spends more time managing shard metadata than indexing data. Too few and you cannot distribute load or recover from node failures in a reasonable time window.

Here is the math most people skip. Each shard is a Lucene instance with its own heap overhead — roughly 1MB per shard for metadata plus per-segment structures. A cluster with 5,000 shards burns around 5GB of heap just on shard bookkeeping before indexing a single document. Elasticsearch's hard limit is 1,000 shards per node, but the practical ceiling is closer to 20 shards per GB of heap on data nodes. A node with 30GB of heap can handle around 600 shards before GC pressure from shard metadata starts affecting search latency.

The target shard size for log workloads is 10GB to 50GB. Below 10GB you are paying overhead on shards too small to benefit from parallelism. Above 50GB, shard recovery after a node failure requires copying the entire shard to a replacement node — a 100GB shard on a 1Gbps network takes around 13 minutes to recover during which that data has one fewer replica. For a daily index receiving 30GB of logs, one primary shard is fine. For 150GB per day, use 5 primary shards.

You cannot change the number of primary shards on an existing index without reindexing. This is one of the most painful lessons in Elasticsearch operations and it is avoidable if you plan before the first document arrives. Use index templates with the correct shard count set before the cluster receives any data. If you get it wrong, reindex is the only path forward — which is a significant operational event.

Replica shards serve two purposes: fault tolerance and read parallelism. For logs, one replica is usually enough — it doubles disk usage but protects against single node failure. If you have 3 data nodes, each primary shard has its primary on one node and its replica on another, giving you tolerance for a single node going down. With 2 replicas across 3 nodes, you have triple the disk usage but can lose any 2 nodes and still serve reads. Choose based on your actual availability requirements, not aspirational ones.

Post-node-restart shard allocation can itself become a cluster bottleneck when shard counts are high. The cluster manager thread handles all allocation decisions, and thousands of pending allocations after a node restart can keep the cluster in a recovering state for much longer than the actual data copy time. Monitor _cluster/allocation/explain when shards are not allocating — it gives the specific reason, from disk watermark to node attribute filtering to replica placement rules. This API saves hours of guessing.

#!/bin/bash
# Elasticsearch shard capacity planning and monitoring commands
# Run these before creating a new index, not after you discover a problem

echo "=== Current shard distribution across data nodes ==="
curl -s 'localhost:9200/_cat/allocation?v&h=node,shards,disk.indices,disk.used,disk.avail,disk.percent'

echo ""
echo "=== Shards over 50GB — candidates for re-sharding on next reindex ==="
curl -s 'localhost:9200/_cat/shards?v&h=index,shard,prirep,store,node' | \
  awk 'NR==1 || ($4 ~ /gb/ && substr($4,1,length($4)-2)+0 > 50)'

echo ""
echo "=== Per-node shard count vs recommended maximum ==="
# Rule: keep shards per node under (heap_GB x 20)
# For a 30GB heap node, maximum is ~600 shards
curl -s 'localhost:9200/_cat/nodes?v&h=name,heapMax,shards' | \
  awk 'NR==1 { print; next }
       {
         heap=$2
         shards=$3
         # Extract numeric heap value (strip units)
         gsub(/[^0-9.]/,"",heap)
         max_shards = heap * 20
         if (shards > max_shards)
           printf "WARNING: %s has %s shards, max recommended %s\n", $1, shards, max_shards
         else
           printf "OK:      %s has %s shards (max %s)\n", $1, shards, max_shards
       }'

echo ""
echo "=== Shard count calculator for a new daily index ==="
DAILY_GB=${1:-50}         # Pass daily volume as first argument, default 50GB
TARGET_SHARD_GB=30        # Target 10-50GB per shard
SHARDS=$(( (DAILY_GB + TARGET_SHARD_GB - 1) / TARGET_SHARD_GB ))  # Ceiling division
echo "Daily volume: ${DAILY_GB}GB"
echo "Target shard size: ${TARGET_SHARD_GB}GB"
echo "Recommended primary shards: ${SHARDS}"
echo "Total disk with 1 replica: $(( DAILY_GB * 2 )) GB (before segment overhead)"
echo "Total disk with 1 replica + 20% overhead: $(echo "$DAILY_GB * 2 * 1.2" | bc)GB"

echo ""
echo "=== Creating index template with calculated shard count ==="
curl -s -XPUT 'localhost:9200/_index_template/app-logs' \
  -H 'Content-Type: application/json' \
  -d "{
    \"index_patterns\": [\"app-logs-*\"],
    \"template\": {
      \"settings\": {
        \"number_of_shards\": ${SHARDS},
        \"number_of_replicas\": 1,
        \"refresh_interval\": \"5s\",
        \"codec\": \"best_compression\",
        \"mapping\": { \"dynamic\": \"strict\" }
      }
    }
  }"

Index Lifecycle Management — Automate Retention Before It Bites You

Without ILM, your ELK stack suffocates under its own data. Daily indices accumulate, disk fills, and someone is running curl commands at 2am to delete old indices. Index Lifecycle Management automates this: you define policies that transition indices through hot, warm, cold, and delete phases based on age, size, or document count. ILM is not optional infrastructure. It is the difference between a cluster that manages itself and one that requires constant manual intervention.

Hot phase: indices are actively written and frequently searched. Keep this short — 1 to 3 days for most log workloads. Use fast NVMe SSDs. This is your most expensive storage tier, and every day you keep an index here costs more than it should.

Warm phase: no more writes, still searchable but with lower urgency. Force-merge to 1 segment — this consolidates all the small segments from active indexing into one, reducing heap overhead and improving scan performance. Before force-merging, Elasticsearch requires the index to have 0 replicas for the shrink operation, which is why the allocate action reducing replicas must precede the shrink action in the policy. Missing this step causes the warm phase to fail silently.

Cold phase: read-only, reduced replica count, optionally migrated to slower storage using data tiers. For compliance-driven retention requirements, this phase can extend to months or years.

Delete phase: remove indices after the retention period. Set this based on your data retention policy and test it with a very short window on a development cluster — 1 hour delete — before using real durations in production.

ILM rollover is the mechanism that prevents any single index from growing beyond your target shard size. When an index exceeds max_size or max_age, ILM rolls over to a new index with the same template settings. This keeps shard sizes predictable and recovery times bounded. Set rollover on both a size trigger and an age trigger — whichever fires first — so that low-volume days still rotate on schedule.

A critical operational detail: ILM policies attached to an index template apply to new indices only. Indices created before the policy was attached require explicit opt-in via the PUT /index/_settings API. And always test your ILM policy on a development cluster before applying it to production. A misconfigured delete phase that fires too early is an availability incident, not a configuration mistake.

// ILM policy: hot 3 days, warm 14 days, delete after 30 days
// PUT _ilm/policy/logs-ilm-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",  // Roll over when index hits 50GB regardless of age
            "max_age":  "1d"     // Also roll over daily even if under 50GB
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          // Reduce replicas to 0 BEFORE shrink — ES requires this for the shrink operation
          // Shrink needs all shards on a single node, which requires no replica competing for placement
          "allocate": { "number_of_replicas": 0 },
          "forcemerge": { "max_num_segments": 1 },
          "shrink":    { "number_of_shards": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

// Attach to index template — new indices inherit this policy automatically
// PUT _index_template/app-logs
{
  "index_patterns": ["app-logs-*"],
  "template": {
    "settings": {
      "index.lifecycle.name":             "logs-ilm-policy",
      "index.lifecycle.rollover_alias":    "app-logs",
      "number_of_shards":                  2,
      "number_of_replicas":                1,
      "mapping": { "dynamic": "strict" }
    }
  }
}

// Verify ILM is working on existing indices:
// GET app-logs-*/_ilm/explain

Cluster Sizing and Hardware Selection — CPU, RAM, Disk Trade-offs

Choosing hardware for Elasticsearch is a trade-off between three resources that compete against each other. The right mix depends entirely on your workload profile — and the profile changes as your data grows.

RAM is the highest-leverage resource. Elasticsearch uses the OS filesystem cache as aggressively as the JVM heap. If your hot index fits in the filesystem cache, searches are nearly instant. If it does not, every search requires disk reads. The practical rule: allocate 50% of node RAM to the JVM heap, leave the rest for the OS. A 64GB node gets 31GB of heap for Elasticsearch and 33GB for the OS cache. Do not exceed 31GB of heap on a single node — above this threshold, the JVM switches from 4-byte compressed object pointers to 8-byte uncompressed ones, which doubles reference size and increases GC pressure. On Elasticsearch 8.x running JDK 21 with generational ZGC, the GC characteristics improve significantly over older G1GC configurations, but the 31GB ceiling on heap remains the safe practical limit. If you need more memory, add nodes rather than increasing per-node heap.

CPU matters most for indexing throughput and complex aggregations. Logstash with heavy Grok patterns can saturate CPU before Elasticsearch does. For data nodes, modern CPUs with high single-threaded clock speeds (3.5GHz or above) benefit search latency on sequential segment scans. Higher core counts improve bulk indexing throughput but have diminishing returns beyond 16 to 20 cores per node for typical log workloads.

Disk is where most teams underprovision. NVMe SSDs are non-negotiable for hot phase data nodes — the random I/O pattern that Elasticsearch generates during segment merges and concurrent searches will saturate spinning disks and cause indexing pauses. For warm and cold phases, SATA SSDs provide acceptable throughput at lower cost. In AWS, gp3 EBS volumes provide 3,000 IOPS baseline with 16,000 IOPS available at lower cost than gp2 — use gp3 for data nodes. io2 is rarely justified for log workloads.

Network is frequently the bottleneck that nobody planned for. Elasticsearch shuffles large amounts of data during shard recovery, rebalancing, and snapshot creation. On a 3-node cluster recovering a 200GB shard after a node replacement, 10GbE networking copies the shard in roughly 3 minutes. On 1GbE, that is 27 minutes during which the data has no replica. Use 10GbE or better between data nodes. In cloud environments, verify that your instance type provides dedicated network bandwidth — burstable instance types that share network bandwidth will throttle under sustained replication load.

Master nodes deserve dedicated resources. Three master-eligible nodes minimum for quorum. Give them 8GB RAM and 4 CPU cores — they manage cluster state, not data, so resource requirements are modest. Never run master-eligible and data roles on the same node in production. A GC pause on a data node caused by heavy indexing can delay master heartbeats, trigger unnecessary master elections, and destabilize the cluster at the worst possible moment. Keep the roles separated.

#!/bin/bash
# Cluster resource health check — run this daily as part of ops review

echo "=== Node roles, heap pressure, and disk usage ==="
curl -s 'localhost:9200/_cat/nodes?v&h=name,node.role,heap.percent,ram.percent,cpu,disk.used_percent'

echo ""
echo "=== JVM heap used percent per node ==="
curl -s 'localhost:9200/_nodes/stats/jvm?pretty' | \
  python3 -c "
import json, sys
nodes = json.load(sys.stdin)['nodes']
for nid, n in nodes.items():
    print(f"{n['name']:20s} heap: {n['jvm']['mem']['heap_used_percent']}%")
"

echo ""
echo "=== Thread pool rejections — indicates resource pressure ==="
curl -s 'localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected' | \
  awk 'NR==1 || $5+0 > 0'  # Show header and any row with rejections > 0

echo ""
echo "=== OS filesystem cache available per node ==="
curl -s 'localhost:9200/_nodes/stats/os?pretty' | \
  python3 -c "
import json, sys
nodes = json.load(sys.stdin)['nodes']
for nid, n in nodes.items():
    mem = n['os']['mem']
    free_gb = mem['free_in_bytes'] / 1024**3
    total_gb = mem['total_in_bytes'] / 1024**3
    print(f"{n['name']:20s} OS cache available: {free_gb:.1f}GB / {total_gb:.1f}GB")
"