Elasticsearch Basics

Elasticsearch is a distributed search and analytics engine built on Apache Lucene that makes large volumes of unstructured data instantly searchable. It powers critical production systems like log analytics, e-commerce search, and observability pipelines at scale. Understanding how its inverted index, mappings, and aggregations work is essential for building fast, reliable search — and for avoiding the data loss and performance failures that come from treating it like a general-purpose database.

What Elasticsearch Actually Does

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. It indexes JSON documents into inverted indices, enabling near-real-time full-text search, structured queries, and aggregations across terabytes of data. The core mechanic is sharding: each index is split into primary and replica shards, distributed across a cluster for parallelism and fault tolerance.

Documents are stored in indices, each shard is a complete Lucene index. Writes go to primary shards, then replicate; reads can hit any replica. This gives you sub-second search on billions of documents — but only if you design your mapping upfront. Dynamic mapping is convenient in dev; in production it causes field explosion, index bloat, and silent query failures.

Use Elasticsearch when you need free-text search, log analytics, or real-time aggregations on semi-structured data. It is not a primary data store — no transactions, no strong consistency guarantees. Teams that treat it as a general-purpose database hit split-brain scenarios, data loss on cluster splits, and painful reindexing. It excels as a secondary index or analytics layer behind a durable source of truth.

Documents and Indices

An index is like a database table but stores JSON documents. Documents are automatically given a unique _id (or you can assign one). The mapping defines the schema (field types, analysers). Unlike SQL, the schema can be dynamic — new fields are automatically added if dynamic mapping is enabled. But dynamic mapping is dangerous in production (see incident above). Each index is composed of one or more shards that are distributed across the cluster. That's how Elasticsearch scales horizontally.

# Index a document (HTTP PUT/POST to the REST API)
PUT /articles/_doc/1
{
  "title": "Understanding Big O Notation",
  "content": "Big O describes the rate of growth of an algorithm...",
  "author": "Alice Chen",
  "published": "2025-03-17",
  "tags": ["algorithms", "computer science", "performance"]
}

# Response:
# { "_index": "articles", "_id": "1", "result": "created" }

# Get document by ID
GET /articles/_doc/1

# Delete document
DELETE /articles/_doc/1

# Bulk indexing (much faster than individual requests)
POST /_bulk
{ "index": { "_index": "articles", "_id": "2" } }
{ "title": "AVL Trees", "content": "Self-balancing BST..." }
{ "index": { "_index": "articles", "_id": "3" } }
{ "title": "Huffman Coding", "content": "Variable-length encoding..." }

Full-Text Search Queries

Elasticsearch's killer feature is full-text search. When you send a match query, Elasticsearch analyses the query string (tokenises, lowercases, applies stemmers) and compares it against the analysed text in the inverted index. Relevance scoring (TF-IDF or BM25) ranks results. You can boost fields, require clauses, and exclude terms using the bool query. multi_match lets you search across multiple fields with individual boosts.

# match query: full-text search with relevance scoring
GET /articles/_search
{
  "query": {
    "match": {
      "content": "algorithm performance"
    }
  }
}

# multi_match: search across multiple fields
GET /articles/_search
{
  "query": {
    "multi_match": {
      "query": "binary tree",
      "fields": ["title^2", "content", "tags"]  // title weighted 2x
    }
  }
}

# bool query: combine must, should, must_not
GET /articles/_search
{
  "query": {
    "bool": {
      "must":     [ { "match": { "content": "sorting" } } ],
      "should":   [ { "match": { "tags": "algorithms" } } ],
      "must_not": [ { "match": { "title": "deprecated" } } ],
      "filter":   [ { "range": { "published": { "gte": "2024-01-01" } } } ]
    }
  }
}

How the Inverted Index Powers Search

The inverted index is what makes full-text search possible without a full scan. When you index a text field, the analyser produces a list of tokens. For every unique token, the inverted index stores a sorted list of document IDs that contain that token, plus the position(s) within the document. When a match query arrives, Elasticsearch looks up each token in the inverted index — that's a hash table lookup, not a linear scan. The search returns the document IDs, scores them, and retrieves the top hits. The process is the same whether you have 1,000 or 100,000,000 documents — search latency scales with the number of distinct tokens, not the document count. That's why Elasticsearch can respond in milliseconds on huge datasets.

# Test how a text field gets analysed (produces tokens)
POST /_analyze
{
  "analyzer": "standard",
  "text": "Elasticsearch is a search engine"
}

# Response tokens:
# ["elasticsearch", "is", "a", "search", "engine"]

# For each token, the inverted index stores doc IDs
# Token "search" -> [doc1, doc5, doc9]
# Token "engine" -> [doc1, doc3]

Full-Text vs Keyword: When to Use Each

Elasticsearch offers two fundamental ways to handle string fields: text (analysed, full-text) and keyword (exact, unanalysed). Choosing the wrong type leads to missing results or broken filters. A text field uses an analyser to break the string into tokens; a keyword field stores the entire string as a single token. Use text for human-readable content you want to search by relevance (blog posts, product descriptions). Use keyword for identifiers, enums, tags, emails, URLs — anything you filter or aggregate on. In many production schemas, the same string field is mapped as both: a text sub-field for match queries and a keyword sub-field for term and aggregations.

# Multi-field mapping: title is both text and keyword
PUT /products/_mapping
{
  "properties": {
    "product_name": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    }
  }
}

# Full-text search on product_name
GET /products/_search
{ "query": { "match": { "product_name": "wireless mouse" } } }

# Exact filter on product_name.keyword
GET /products/_search
{ "query": { "term": { "product_name.keyword": "Wireless Mouse MX Master 3" } } }

# Aggregation on the keyword field
GET /products/_search
{
  "size": 0,
  "aggs": {
    "brands": {
      "terms": { "field": "product_name.keyword" }
    }
  }
}

Aggregations — Analytics

Aggregations let you compute analytics over search results without writing custom code. They're like SQL GROUP BY on steroids: nested aggregations, date histograms, percentile ranks, geo distance, and more. Aggregations run on the shards in parallel and results are merged on the coordinating node. The size: 0 in the request avoids returning documents when you only want the aggregated data.

# Count articles per author and average title length
GET /articles/_search
{
  "size": 0,  // don't return documents, only aggregation results
  "aggs": {
    "by_author": {
      "terms": { "field": "author.keyword", "size": 10 },
      "aggs": {
        "avg_content_length": {
          "avg": { "field": "content_length" }
        }
      }
    },
    "articles_over_time": {
      "date_histogram": {
        "field": "published",
        "calendar_interval": "month"
      }
    }
  }
}

Mapping, Analysis and Tokenisation

Mapping defines how documents and their fields are stored and indexed. Analysis is the process of converting a text field into tokens (terms) that go into the inverted index. It consists of a character filter, a tokeniser, and token filters. By default, Elasticsearch uses the standard analyser (lowercases, splits on punctuation, removes common words). You can create custom analysers with different tokenisers (whitespace, keyword, n-gram) and filters (synonym, stemmer, stop). Understanding analysis is critical to getting search results that match user intent.

# Create an index with a custom analyser
PUT /my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_custom_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "unique"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "my_custom_analyzer"
      }
    }
  }
}

# Test the analyser
POST /my_index/_analyze
{
  "analyzer": "my_custom_analyzer",
  "text": "Elasticsearch Search Engine"
}
# Result: tokens => ["elasticsearch", "search", "engine"]

Mapping & Analyzers: Technical Reference Guide

This reference covers the essential mapping types, parameter configuration, and custom analyser components you'll use in production. The mapping is the schema definition for an index — it tells Elasticsearch how to store and index each field. Key field types: text (full-text), keyword (exact match), integer, long, float, double, boolean, date (ISO 8601), nested (array of objects), flattened (arbitrary key-value), geo_point, ip. Each type has parameters: index (true/false), store (separate stored field), doc_values (for aggregations/sorting), norms (scoring), copy_to (combine fields).

Analysers consist of: character filters (strip HTML, replace patterns), tokeniser (standard, whitespace, edge_ngram), token filters (lowercase, stemmer, stop, synonym, asciifolding). Custom analysers are defined in index settings and referenced in mappings.

Production-critical parameters: ignore_above (keyword length limit), coerce (auto-convert numbers), eager_global_ordinals (pre-load for high cardinality keyword fields used in aggregations), fields (multi-field for same string).

# Full-featured mapping with custom analyser
PUT /logs/_mapping
{
  "dynamic": false,
  "properties": {
    "message": {
      "type": "text",
      "analyzer": "my_standard_analyzer",
      "fields": {
        "keyword": { "type": "keyword", "ignore_above": 512 },
        "ngram": { "type": "text", "analyzer": "ngram_analyzer" }
      }
    },
    "severity": {
      "type": "keyword",
      "doc_values": false,  # not aggregated
      "norms": false         # no scoring needed
    },
    "timestamp": { "type": "date" }
  },
  "settings": {
    "index.mapping.total_fields.limit": 200,
    "analysis": {
      "char_filter": {
        "html_strip": { "type": "html_strip" }
      },
      "tokenizer": {
        "edge_ngram": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 10
        }
      },
      "filter": {
        "my_synonyms": {
          "type": "synonym",
          "synonyms": ["fast, quick", "small, little"]
        }
      },
      "analyzer": {
        "my_standard_analyzer": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "my_synonyms", "stop"]
        },
        "ngram_analyzer": {
          "type": "custom",
          "tokenizer": "edge_ngram",
          "filter": ["lowercase"]
        }
      }
    }
  }
}

Cluster Architecture, Sharding and Scaling

An Elasticsearch cluster consists of nodes: master-eligible nodes (handle cluster state), data nodes (store data and execute queries), ingest nodes (pre-process documents), and coordinating nodes (route requests). Data is split into shards — each shard is a separate Lucene index. When you index a document, it is routed to a primary shard based on _routing (default: document ID hash). The primary shard synchronously replicates to replica shards on other nodes. Scaling means adding data nodes — shards are automatically rebalanced. But you cannot change the number of primary shards after index creation. So right-sizing shards at index creation time is critical.

# Check cluster health
GET /_cluster/health?pretty
# Sample output: { "status": "green", "number_of_nodes": 5, "active_shards": 20 }

# View shard allocation
GET /_cat/shards?v

# Create index with custom shard count and replication
PUT /my_logs
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 2
  }
}

Shard & Replica Distribution Layout

Visualising how shards and replicas are distributed across nodes helps you understand resilience and read scalability. Each index has a set of primary shards (P0, P1, ...). Each primary shard has zero or more replica shards (R0, R1, ...) on different nodes. Elasticsearch ensures primary and replica shards for the same shard number are never allocated on the same node — this guarantees high availability. When you have replicas, read requests can be served by any copy, distributing load. The diagram below shows a 2-node cluster with an index of 3 primary shards and 1 replica each.

# Create index with 3 primary shards and 1 replica
PUT /my_index
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

# View allocation breakdown
GET _cat/shards/my_index?v&h=index,shard,prirep,state,node
# index   shard prirep state   node
# my_index 0     p      STARTED node-1
# my_index 0     r      STARTED node-2
# my_index 1     p      STARTED node-2
# my_index 1     r      STARTED node-1
# my_index 2     p      STARTED node-1
# my_index 2     r      STARTED node-2

Why Your Elasticsearch Cluster Died at 3 AM — Heap Pressure and GC Pauses

Every production meltdown I’ve seen with Elasticsearch traces back to one thing: the JVM heap. It’s not disk space. It’s not network. It’s the garbage collector stalling because you treated Elasticsearch like a document store instead of a search engine.

Elasticsearch runs on Lucene. Lucene writes to segments on disk, not to heap. But your aggregations, parent-child relationships, and heavy terms queries build in-memory hash tables that eat young gen alive. When those promotions fail, you get long GC pauses. Nodes drop out of the cluster. Shards go missing. You wake up to a 503.

The fix starts before deployment. Set -Xmx to no more than 50% of available RAM. The rest goes to the OS page cache — that’s where Lucene actually lives. Use _cat/nodes?v&h=heap.percent,ram.percent to monitor the split. If heap percent stays above 85 under load, you’re trading speed for stability. Split the index, cache fewer fields, or push aggregations to dedicated coordinating nodes.

// io.thecodeforge — database tutorial

// Check per-node heap and RAM usage across cluster
GET _cat/nodes?v&h=name,node.role,heap.current,heap.percent,ram.percent

// Sample output:
// name      node.role heap.current heap.percent ram.percent
// es-data-1 di         12.8gb       82           91
// es-data-2 di         14.1gb       94           88
// es-master m          2.1gb        22           45

// If heap.percent > 85 for data nodes:
// 1. Reduce shard count per node (max 20 per GB heap)
// 2. Disable fielddata on text fields you don't need
PUT /_cluster/settings
{
  "transient": {
    "indices.fielddata.cache.size": "20%"
  }
}

The Write Path — Why Your Indexing Throughput Is Lying to You

You watch the indexing rate on the monitoring page and think everything’s fine. Then you do a forced refresh and all those buffered writes flood the disk. Your search latency spikes. Users complain. You had 10k docs/second incoming but the disk queue depth hit 100. That’s not throughput. That’s deferred pain.

Elasticsearch doesn’t write to disk on every _bulk request. It keeps segments in memory until a refresh (every second by default) or a flush (to translog). That translog is your safety net if the node crashes — but replaying it on startup can take minutes if you let it grow. A shard with 1 million uncommitted docs after an unclean shutdown? Good luck.

The move: use index.translog.sync_interval to control flush frequency. For bulk-heavy pipelines, set it to 30s and live with a slightly larger recovery window. Never use _forcemerge until indexing is fully done — it rewrites segments and destroys the file system cache. Batch your writes. Keep segment count under 150 per shard. And for the love of god, disable replicas during reindexing and turn them back on after.

// io.thecodeforge — database tutorial

// Optimize index for high-volume writes (bulk load only)
PUT /logs_write_optimized
{
  "settings": {
    "index": {
      "refresh_interval": "-1",           // turn off auto-refresh
      "number_of_replicas": 0,            // no replicas during load
      "translog": {
        "sync_interval": "30s",
        "durability": "async"            // safer: 'request' for durability
      },
      "segments": {
        "total_fields": 2000
      }
    }
  }
}

// After all data indexed:
PUT /logs_write_optimized/_settings
{
  "index": {
    "refresh_interval": "30s",
    "number_of_replicas": 1
  }
}
POST /logs_write_optimized/_forcemerge?max_num_segments=5

Setup and Installation — Get Elasticsearch Running Before You Burn an Hour

Most engineers waste time fighting installation when they should be fighting bad queries. Elasticsearch runs on Java, so your first check is the JVM version. OpenJDK 17 or 21. No exceptions. Download the tarball from Elastic's official site — neverapt or brew outside of dev. Untar, run bin/elasticsearch, and it boots on port 9200. If you want Docker, use the official image with explicit memory limits: -Xms1g -Xmx1g. Defaults will eat your laptop alive. Production installs need the elasticsearch.yml file tuned for your cluster. Set discovery.type: single-node for local testing, then discovery.seed_hosts for real setups. The data and logs directories must be on separate volumes. Don't run as root — create a dedicated elasticsearch user. Verify with curl -XGET 'localhost:9200'. If you see a cluster name, you're live. If you see an error, check your heap. Always.

// io.thecodeforge — database tutorial

-- Download and extract Elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.12.0-linux-x86_64.tar.gz
tar -xzf elasticsearch-8.12.0-linux-x86_64.tar.gz
cd elasticsearch-8.12.0

-- Configure minimum production settings
echo "cluster.name: prod-cluster" >> config/elasticsearch.yml
echo "node.name: node-1" >> config/elasticsearch.yml
echo "path.data: /var/lib/elasticsearch" >> config/elasticsearch.yml
echo "path.logs: /var/log/elasticsearch" >> config/elasticsearch.yml
echo "discovery.seed_hosts: [\"10.0.0.1\", \"10.0.0.2\"]" >> config/elasticsearch.yml

-- Start Elasticsearch (non-root user recommended)
useradd -m elasticsearch
chown -R elasticsearch:elasticsearch /opt/elasticsearch-8.12.0
su elasticsearch -c './bin/elasticsearch -d -p pid'

-- Health check
curl -XGET 'localhost:9200/_cluster/health?pretty'

Security and Access Control — Stop Running Elasticsearch With No Pants

Out of the box, Elasticsearch listens on 0.0.0.0:9200 with no authentication. That's a lawsuit waiting to happen. The first thing you do is enable the Elasticsearch security features. In version 8+, it's automatic — but only if you didn't override it. For older clusters, set xpack.security.enabled: true in elasticsearch.yml. Then run bin/elasticsearch-setup-passwords auto to generate credentials for the elastic superuser. Store those in a vault, not a sticky note. Next, create roles per use case: read_only for dashboards, ingest for Logstash, admin for your CI. Use the security API, not the config file. TLS is non-negotiable for transport between nodes. Generate certificates with bin/elasticsearch-certutil cert. If you're on Kubernetes, use the Elasticsearch operator — it handles secrets and certs for you. API keys are preferred over passwords for automation. Revoke them with one DELETE call. Audit logs catch the intern who drops an index at 2 AM. Turn them on with xpack.security.audit.enabled: true.

// io.thecodeforge — database tutorial

-- Enable security in config
echo "xpack.security.enabled: true" >> config/elasticsearch.yml
echo "xpack.security.transport.ssl.enabled: true" >> config/elasticsearch.yml
echo "xpack.security.transport.ssl.verification_mode: certificate" >> config/elasticsearch.yml

-- Generate TLS certificates for inter-node communication
bin/elasticsearch-certutil cert --pem --out tls-certs.zip

-- Create a read-only role via API
curl -XPOST 'localhost:9200/_security/role/read_only' \
  -H 'Content-Type: application/json' \
  -u elastic:YOUR_PASSWORD \
  -d '{
    "cluster": ["monitor"],
    "indices": [{
      "names": ["logs-*"],
      "privileges": ["read", "view_index_metadata"]
    }]
  }'

Why Data Ingestion Kills Your Cluster — The Ingestion Pipeline

Elasticsearch is only as fast as the data you feed it. Most engineers skip the ingestion pipeline and wonder why indexing blocks, field mappings drift, or documents disappear. The why: raw data from logs, APIs, or databases arrives in messy shapes. Elasticsearch can't search what it can't parse. The how: use ingest nodes with built-in processors — grok for unstructured logs, date index for timestamps, and set/remove for field cleanup. A pipeline runs before indexing, transforming each document. This prevents mapping explosions, reduces storage by dropping null fields, and improves query speed by pre-computing derived values. The production trap: without a pipeline, every source change requires reindexing your entire cluster. Pipeline versioning lets you evolve schemas without downtime. Key rule: always attach a pipeline to your index template — it's your data's first line of defense.

// io.thecodeforge — database tutorial

PUT _ingest/pipeline/log_cleanup
{
  "description": "Clean raw logs before indexing",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{COMBINEDAPACHELOG}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["ISO8601"]
      }
    },
    {
      "remove": {
        "field": ["extra_headers", "raw_bytes"]
      }
    }
  ]
}

Advanced Querying — Why bool Queries Crush Your Lucene Score

// io.thecodeforge — database tutorial

GET /logs/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "error" } }
      ],
      "filter": [
        { "term": { "status": 500 } },
        { "range": { "@timestamp": { "gte": "now-7d" } } }
      ],
      "should": [
        { "match": { "severity": "critical" } }
      ],
      "minimum_should_match": 1
    }
  }
}

Prerequisites to Learn Elasticsearch

Before building anything useful with Elasticsearch, you need a solid grasp of RESTful APIs (curl or Postman) and JSON structure, because every operation from indexing to search is a JSON-over-HTTP call. You must understand your data’s shape — fields that will be searched, aggregated, or sorted — so you can define a proper mapping. Without that, Elasticsearch will infer types, often disastrously (e.g., treating a ZIP code as a numeric field). You should also know the difference between structured (numeric, date) and unstructured (full-text) data; they require different analyzers. Cluster concepts matter: a single-node setup hides shard/replica behavior that will bite you in production. Finally, install Java 17+ (or the version required by your Elasticsearch release). A common pain point: default JVM heap settings (1 GB) cause immediate OutOfMemoryErrors when indexing even modest datasets. Tune heap to 50% of available RAM, not more than 32 GB, to avoid compressed object pointer overhead. Start small: index a few documents, run a match query, then scale to bulk ingestion.

// io.thecodeforge — database tutorial
// Verify prerequisites: heap size and node roles
GET _cat/nodes?v&h=name,heap.current,heap.percent,node.role

// Expected output shows heap usage <75%
// and node.role includes 'm' (master-eligible)
// If heap.percent >85, resize heap immediately.
// curl localhost:9200/_cat/nodes?v&h=name,heap*

Comparisons and Differences

Elasticsearch is often confused with relational databases (MySQL, PostgreSQL) and other search engines (Solr). The fundamental difference: ES is a distributed, near-real-time document store with inverted index architecture, not a structured table. Joins are expensive — ES prefers denormalized, nested, or parent-child mappings. Unlike traditional DBs, ES has no ACID transactions across shards; it favors eventual consistency for speed. Compare with Solr: Solr has better built-in faceting for e-commerce, but ES wins on ecosystem (Kibana, Beats, Logstash) and horizontal scalability driven by Zen Discovery. Against OpenSearch? It's a fork; performance and API are nearly identical, but ES has a larger community and more aggressive feature development. The biggest difference from any SQL database: ES does not support multi-row transactions or foreign keys. If you need strong consistency, pair ES with a source-of-truth DB and treat it as a secondary index. For time-series data (logs, metrics), ES excels over MongoDB because of inverted indices and aggregation pipelines. Choose ES when you need full-text search, analytics dashboards, or high-ingestion-rate logging — not for banking ledgers.

// io.thecodeforge — database tutorial
// ES match query vs SQL LIKE — fundamentally different
GET products/_search
{
  "query": {
    "match": {
      "description": "running shoes"
    }
  }
}

// Equivalent SQL would be:
// SELECT * FROM products WHERE description LIKE '%running shoes%';
// SQL is sequential scan; ES uses inverted index scoring.
// ES returns relevance rankings; SQL returns exact substring.

Conclusion

Elasticsearch is a powerful tool, but it demands respect for its distributed nature and memory-sensitive internals. The key takeaway from this series: mapping and analysis define how your data is stored and searched — get them wrong, and you'll chase phantom relevance issues. Cluster architecture, shard distribution, and heap tuning are not optional; they determine whether your cluster survives a traffic spike or implodes at 3 AM. The write path and ingestion pipelines are the primary bottlenecks — always measure throughput with bulk requests and monitor GC logs. Security, often an afterthought, should be configured before the first document is indexed (enable TLS, set password for elastic user). Comparisons with SQL databases or Solr clarify use cases: ES excels at full-text search and log analytics, not relational consistency. Finally, prerequisites like REST fluency and data-type awareness form the foundation; skip them and you'll wrestle with inferred mappings that corrupt search accuracy. Start small, monitor relentlessly, and treat every mapping change as a migration. Elasticsearch rewards understanding — it punishes blind optimism. Use the provided code snippets as daily checklists, and you'll push to production with confidence.

// io.thecodeforge — database tutorial
// Pre-deployment checklist — must run all
GET _cluster/health?wait_for_status=green&timeout=30s
GET _cat/allocation?v&h=shards,disk.percent,ip
GET _nodes/stats/jvm?filter_path=nodes.*.jvm.gc.collectors.young.collection_time_in_millis

// Expected:
// - status: green (all shards active)
// - disk.percent < 80 on every node
// - young GC time < 10s per hour
// If any check fails, stop and fix.