Spring Data Elasticsearch: Full-Text Search and Analytics in Spring Boot 3.2

In the world of real-time analytics and search, a traditional relational database often falls short. When you need to search millions of documents in milliseconds, aggregate data on the fly, or handle full-text search with fuzzy matching, Elasticsearch becomes your go-to. Spring Data Elasticsearch bridges the gap between your Spring Boot application and Elasticsearch clusters, providing a familiar repository pattern while exposing the raw power of Elasticsearch queries.

I've been building search-heavy applications for over a decade. In 2016, I worked on a SaaS billing platform that required instant search across 50 million invoices. We started with PostgreSQL full-text search, which worked for a few thousand records, but by the time we hit a million, queries took seconds. We migrated to Elasticsearch 2.x with Spring Data Elasticsearch, and search latency dropped to under 50ms. That experience taught me the importance of proper index mapping, query optimization, and cluster sizing.

This tutorial assumes you have a working Spring Boot 3.2 application and an Elasticsearch 8.11 instance (local or Docker). We'll cover entity mapping, repository creation, full-text search, aggregations, and production debugging. By the end, you'll be able to build a search and analytics layer that can handle millions of documents with sub-second response times. We'll also dive into the pitfalls I've seen in production — like index mapping conflicts that cause silent data loss, and the infamous "too_many_clauses" error that can bring down your cluster.

Setting Up Spring Data Elasticsearch

First, add the required dependencies to your pom.xml. For Spring Boot 3.2, use the spring-boot-starter-data-elasticsearch starter. I recommend using the latest version to avoid compatibility issues. In one of my projects, we used an older version and spent two days debugging a NoClassDefFoundError because of conflicting Elasticsearch client versions. Stick with 5.1.x for Spring Boot 3.2.

Next, configure the connection to your Elasticsearch cluster in application.properties. For local development, you can use http://localhost:9200. In production, you'll want to enable SSL and authentication. The example below shows a basic setup.

Create an entity class annotated with @Document to map to an Elasticsearch index. The @Field annotation allows you to specify the type and analyzer for each field. For full-text search, use FieldType.Text with a standard analyzer. For aggregations and sorting, use FieldType.Keyword.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
    <version>3.2.0</version>
</dependency>

What the Official Docs Won't Tell You

The official Spring Data Elasticsearch documentation is decent, but it glosses over several critical details that will bite you in production. First, the @Document annotation's createIndex attribute defaults to true, which means Spring will try to create the index on startup. This is fine for development, but in production, you should set it to false and manage index creation manually. I've seen applications crash because the application user didn't have index creation permissions.

Second, the @Field annotation's analyzer attribute is ignored if you don't also set searchAnalyzer. The analyzer is used for indexing, and the searchAnalyzer is used for querying. If you only set the analyzer, your search queries might use a different analyzer, leading to unexpected results. For example, if you use a custom analyzer with synonym support, but the search analyzer defaults to the standard analyzer, synonyms won't work.

Third, the Page interface in Spring Data Elasticsearch does not support total hit count accurately when using aggregations. The getTotalElements() method returns the total number of documents matching the query, not the aggregation bucket count. If you need both pagination and aggregations, you must use NativeSearchQuery and handle the response manually.

Finally, the @Id field must be of type String or Long. Using UUID or other types can cause serialization issues. I once spent an afternoon debugging why a document wasn't updating — the ID was a UUID, and Elasticsearch was converting it to a string differently than Spring expected.

spring.elasticsearch.uris=http://localhost:9200
spring.elasticsearch.connection-timeout=5s
spring.elasticsearch.socket-timeout=30s
# In production, use:
# spring.elasticsearch.uris=https://your-cluster:9200
# spring.elasticsearch.username=your-user
# spring.elasticsearch.password=your-password

Entity Mapping and Index Configuration

Mapping your Java entity to an Elasticsearch index is straightforward, but there are nuances. The @Document annotation defines the index name, shards, and replicas. For a payment processing system, you might have an index per month (e.g., payments-2024-01) to manage data lifecycle. The shards and replicas settings are crucial for performance and high availability. In production, I use 3 shards and 2 replicas for most indices.

The @Field annotation is where you define the Elasticsearch field type. For full-text search, use FieldType.Text with a custom analyzer if needed. For exact matches and aggregations, use FieldType.Keyword. You can also use FieldType.Date for dates, FieldType.Long for numbers, and FieldType.Object for nested documents.

One common mistake is not using the index attribute. By default, all fields are indexed. If you have fields that are only used for display (not search), set index = false to save disk space and improve indexing speed. In a billing system, we had a description field that was never searched, but was indexed by default. Turning off indexing reduced index size by 20%.

@Document(indexName = "payments-#{@environment.getProperty('elasticsearch.index.suffix')}", shards = 3, replicas = 2)
public class PaymentTransaction {
    @Id
    private String id;

    @Field(type = FieldType.Text, analyzer = "standard", searchAnalyzer = "standard")
    private String description;

    @Field(type = FieldType.Keyword)
    private String status;

    @Field(type = FieldType.Double)
    private Double amount;

    @Field(type = FieldType.Date, format = DateFormat.date_hour_minute_second)
    private LocalDateTime transactionDate;

    @Field(type = FieldType.Keyword, index = false)
    private String internalNotes;

    // getters and setters
}

Creating a Search Repository

Spring Data Elasticsearch provides several repository interfaces. The simplest is ElasticsearchRepository<T, ID>, which gives you CRUD operations and basic search methods. For full-text search, you need SearchHits to get the search metadata like score and highlights.

To create custom search methods, you can use the @Query annotation with a JSON query string. This is powerful but error-prone. I prefer using NativeSearchQueryBuilder for complex queries because it provides type safety and better readability.

One pattern I've used in production is the "search service" that wraps the repository. This service handles pagination, sorting, and aggregation logic. It also catches Elasticsearch exceptions like ElasticsearchStatusException and translates them to meaningful HTTP responses.

For the payment processing example, we need a method to search transactions by description, filter by status, and aggregate by amount ranges. This is where the real power of Elasticsearch shines.

public interface PaymentTransactionRepository extends ElasticsearchRepository<PaymentTransaction, String> {
    
    // Custom query using @Query
    @Query("{\"match\": {\"description\": \"?0\"}}")
    SearchHits<PaymentTransaction> findByDescription(String description);

    // Custom query with aggregation
    @Query("{\"bool\": {\"filter\": [{\"term\": {\"status\": \"?0\"}}]}}")
    SearchHits<PaymentTransaction> findByStatus(String status);
}

Advanced Full-Text Search with NativeSearchQuery

For production-grade search, you need more than simple match queries. You need multi-field search, fuzzy matching, boosting, and highlighting. The NativeSearchQueryBuilder is your Swiss Army knife.

Consider a scenario where users search for "payment failed" and you want to match against both the description and status fields, but give more weight to the description. You can use a bool query with should clauses and boosting. Add fuzzy matching to handle typos like "paymnet".

Highlighting is another critical feature. When displaying search results, you want to show the user why a document matched. Elasticsearch returns highlighted fragments that you can render in the UI. I've used this in a SaaS billing dashboard to show users exactly which part of the invoice description matched their query.

Aggregations are where Elasticsearch truly excels over relational databases. You can compute metrics like total amount by status, average transaction value, or histogram of transactions over time, all in a single query. This eliminates the need for separate analytics endpoints and reduces load on your database.

@Service
public class PaymentSearchService {
    private final ElasticsearchOperations operations;

    public PaymentSearchService(ElasticsearchOperations operations) {
        this.operations = operations;
    }

    public SearchHits<PaymentTransaction> search(String query, int page, int size) {
        NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withQuery(QueryBuilders.boolQuery()
                .should(QueryBuilders.matchQuery("description", query).boost(2.0f))
                .should(QueryBuilders.matchQuery("status", query).fuzziness(Fuzziness.ONE))
            )
            .withPageable(PageRequest.of(page, size))
            .withHighlightBuilder(new HighlightBuilder()
                .field("description")
                .preTags("<em>")
                .postTags("</em>")
            )
            .build();
        return operations.search(searchQuery, PaymentTransaction.class);
    }
}

Aggregations for Real-Time Analytics

Aggregations allow you to compute metrics and build dashboards without additional database queries. In the payment processing domain, you might want to show the total revenue by status, the number of transactions per hour, or the average transaction amount by payment method.

There are two types of aggregations: metric aggregations (e.g., sum, avg, min, max) and bucket aggregations (e.g., terms, date_histogram, range). You can nest aggregations to create complex analytics. For example, you can use a date_histogram to group transactions by hour, and within each hour, use a terms aggregation to break down by status.

One pitfall: aggregations can be expensive on large datasets. Always set the size parameter on terms aggregations to limit the number of buckets. The default is 10, which is usually fine, but if you need more, be cautious. I've seen clusters crash because a terms aggregation returned 100,000 buckets.

Another tip: use cardinality aggregation for unique counts (e.g., distinct customers) instead of loading all documents. It's approximate but fast, and the error rate is typically less than 5%.

@Service
public class PaymentAnalyticsService {
    private final ElasticsearchOperations operations;

    public Map<String, Object> getRevenueByStatus() {
        NativeSearchQuery query = new NativeSearchQueryBuilder()
            .addAggregation(AggregationBuilders.terms("by_status")
                .field("status")
                .subAggregation(AggregationBuilders.sum("total_amount").field("amount"))
            )
            .withPageable(PageRequest.of(0, 1))
            .build();

        SearchHits<PaymentTransaction> hits = operations.search(query, PaymentTransaction.class);
        Aggregations aggregations = hits.getAggregations();
        Terms byStatus = aggregations.get("by_status");
        Map<String, Object> result = new HashMap<>();
        for (Terms.Bucket bucket : byStatus.getBuckets()) {
            Sum sum = bucket.getAggregations().get("total_amount");
            result.put(bucket.getKeyAsString(), sum.getValue());
        }
        return result;
    }
}

Production Debugging and Performance Tuning

In production, Elasticsearch performance can degrade for many reasons: slow queries, inadequate sharding, or cluster congestion. Here are the tools and techniques I've used to debug issues.

First, enable slow query logging in Elasticsearch. This logs queries that take longer than a threshold (e.g., 500ms). You can see the actual query JSON and execution time. In Spring Boot, you can also enable logging for the Elasticsearch client: logging.level.org.springframework.data.elasticsearch.client.trace=DEBUG.

Second, use the Elasticsearch _cat/thread_pool API to monitor search and index threads. If you see high rejection rates, your cluster is overloaded. You might need to increase the number of nodes or reduce the query load.

Third, optimize your queries. Avoid using wildcard queries with leading wildcards (e.g., "*payment") because they cannot use the inverted index and require a full scan. Use match_phrase_prefix instead. Also, avoid large terms queries with thousands of values — they cause the "too_many_clauses" error.

Finally, tune your index settings. The refresh_interval defaults to 1 second. For bulk indexing operations, set it to -1 (disabled) to improve write performance, then re-enable it after indexing. In a batch job that indexed 10 million records, we reduced indexing time from 2 hours to 30 minutes by disabling refresh and replicas during the process.

@Configuration
public class ElasticsearchConfig {
    @Bean
    public ElasticsearchOperations elasticsearchOperations(ElasticsearchClient client) {
        return new ElasticsearchTemplate(client);
    }

    @EventListener(ApplicationReadyEvent.class)
    public void configureIndex() {
        // Disable refresh for bulk operations
        UpdateSettingsRequest request = new UpdateSettingsRequest.Builder()
            .index("payments-*")
            .settings(s -> s.index(i -> i.refreshInterval(t -> t.time("-1"))))
            .build();
        // ... execute with client
    }
}

Handling Production Incidents with Elasticsearch

Elasticsearch incidents are inevitable in production. Here's a real incident I dealt with: the "too_many_clauses" error. Our application allowed users to filter by multiple statuses. The UI sent a list of 200 status values, which translated to a terms query with 200 terms. Elasticsearch has a default limit of 1024 clauses, but our query also had other bool clauses, pushing it over the limit.

The symptom was a 500 error with ElasticsearchStatusException: maxClauseCount is set to 1024. The assumption was that the limit would never be reached. The root cause was that the UI allowed unlimited selection, and a power user selected all available statuses.

The fix was twofold: first, we added server-side validation to limit the number of selected statuses to 50. Second, we increased the maxClauseCount setting in Elasticsearch to 4096, but this is a temporary fix. The better solution was to redesign the filter to use a bool query with should clauses, which has a higher limit.

Another common incident is the "circuit_breaking_exception" when memory usage spikes. This happens when a query or aggregation tries to load too much data into memory. The fix is to optimize the query, reduce the number of shards, or increase the heap size of the Elasticsearch nodes.

@Component
public class SearchRequestValidator {
    private static final int MAX_STATUSES = 50;

    public void validate(SearchRequest request) {
        if (request.getStatuses() != null && request.getStatuses().size() > MAX_STATUSES) {
            throw new IllegalArgumentException("Cannot filter by more than " + MAX_STATUSES + " statuses");
        }
    }
}