Design Google Search — Index Corruption Pitfalls

Google processes over 8.5 billion searches per day. Behind every one of those queries is a pipeline that spans web crawling, link graph analysis, distributed indexing, real-time query parsing, and sub-100ms result serving — all at a scale that dwarfs most technology stacks in existence. When interviewers ask you to design a search engine, they're not expecting you to rebuild PageRank from scratch. They're testing whether you can reason about hard trade-offs: freshness vs. consistency, recall vs. precision, crawl budget vs. coverage. This is where senior engineers separate themselves.

The core problem search solves is deceptively simple: given an unstructured corpus of hundreds of billions of web pages, return the most relevant ten results for an arbitrary natural-language query in under 200 milliseconds. The difficulty is in every word of that sentence — 'hundreds of billions' demands distributed storage, 'arbitrary query' demands linguistic understanding, 'most relevant' demands a ranking model trained on human behavior, and '200 milliseconds' demands aggressive caching, pre-computation, and hardware co-design.

By the end of this article you'll be able to walk into any senior system design interview and articulate the full pipeline — from DNS to rendered SERP — naming the right data structures, explaining the right trade-offs, and spotting the gotchas that trip up candidates who only memorized diagrams. We'll cover the crawler, the indexing pipeline, the serving stack, and the ranking layer, with concrete pseudo-code and real numbers wherever they matter.

The kicker: most engineers describe a linear pipeline and forget the feedback loops. The crawler discovers URLs from the indexer's freshness requirements, the indexer publishes stats that reshape the crawl priority, and the ranking model's performance dictates which segments get reindexed first. If you design each component in isolation, you get a system that's correct on paper but breaks in production. For example, a failing crawler on a high-traffic domain can cause the indexer to serve stale results for weeks before anyone notices — because the monitoring dashboard shows 'healthy' but the freshness metric never gets updated. That's the kind of failure that only surfaces when you instrument every boundary. Master the feedback loops, and you'll walk into any interview knowing exactly what separates a diagram-drawer from a system thinker.

One more thing: when you're designing the feedback loops, instrument every cross-component call with a latency histogram and a success/failure counter. Without that data, you're guessing. And trust me — you don't want to debug a silent crawl starvation at 3 AM on a Friday.

What is Design Google Search?

Design Google Search is a core concept in System Design. It's not about building the actual Google — it's about understanding the architectural principles that let you serve relevant results from a web-scale corpus at sub-200ms latency. The system decomposes into four main layers: a distributed web crawler that discovers and downloads pages, an indexing pipeline that transforms raw HTML into searchable inverted indexes, a query serving stack that processes user queries against the index, and a ranking system that orders results by relevance. Each layer must handle failure gracefully, scale horizontally, and operate under strict latency budgets. Senior engineers focus on the boundaries between these layers: how the crawler feeds the indexer without overwhelming it, how the index is distributed to the serving tier, and how ranking models are trained offline and served online without latency spikes.

Here's the thing most tutorials skip: the real bottleneck isn't the crawler or the indexer — it's the coordination between them. If the indexer falls behind, results go stale. If the crawler outpaces the indexer, you waste storage on pages nobody can query. The production reality is you need backpressure signals between every stage, and those signals are often the hardest thing to get right. Don't assume your pipeline is linear — it's a loop with feedback.

And then there's the failure mode that gets ignored until it bites you: a single slow component in the feedback loop can cascade. For example, if the ranking model's offline training pipeline stalls, new features never get deployed, but you don't notice until CTR drops 10% a week later. You need monitoring on every stage's throughput and latency, not just the serving tier.

Another overlooked feedback loop: the crawl priority influences indexing freshness, and indexing freshness influences ranking relevance. A page that's frequently updated but has low crawl priority will never accumulate enough ranking signals to surface in top results. You need a priority feedback signal from the ranking system back to the crawler — if a page's CTR drops below a threshold, increase its crawl frequency. This closes the loop and prevents stale content from polluting results.

Here's the production truth that's hard to swallow: most search engines fail not because the index is slow, but because the coordination signals between stages are missing. Fix the signals, and the system runs itself.

Don't fall into the trap of thinking each layer is independent. The real system is a set of feedback loops: the crawler feeds the indexer, the indexer's freshness determines ranking, and ranking performance dictates crawl priority. If you design in isolation, you'll miss the failures that happen at the boundaries.

One more thing: when you're designing the feedback loops, instrument every cross-component call with a latency histogram and a success/failure counter. Without that data, you're guessing. And trust me — you don't want to debug a silent crawl starvation at 3 AM on a Friday.

package io.thecodeforge.search;

/**
 * Simplified inverted index lookup — production uses distributed shards.
 */
public class InvertedIndexLookup {
    private final Map<String, List<Posting>> index;

    public InvertedIndexLookup(Map<String, List<Posting>> index) {
        this.index = index;
    }

    public List<Posting> lookup(String term) {
        List<Posting> postings = index.get(term);
        if (postings == null) return List.of();
        // In real system, merge with phrase/positional index
        return postings;
    }

    static class Posting {\n        final int docId;\n        final float score; // e.g., TF-IDF contribution\n        Posting(int docId, float score) {\n            this.docId = docId;\n            this.score = score;\n        }
    }
}

Web Crawling: Discovery, Fetching, and Politeness

The crawler discovers URLs from a seed set and downloads each page's content. At Google scale, you can't crawl everything every day — you need a crawl budget. Popular pages get fetched daily, while less important ones might be re-crawled weekly. The crawler must respect robots.txt, detect duplicate content, and avoid overwhelming servers. A critical component is the URL frontier, a priority queue that decides which URLs to fetch next based on freshness, page importance, and politeness. The crawler uses DNS caching, HTTP persistent connections, and distributed storage (e.g., Bigtable) to store raw HTML. Deduplication via simhash prevents storing near-identical pages.

But here's the part that catches everyone off guard: the frontier itself becomes a distributed system at scale. You can't keep a single priority queue in memory when you have billions of URLs. You need a partitioned frontier where each partition handles a range of domains, and you need a coordinator to balance load. And politeness isn't just about per-domain rate — it's about detecting when a site pushes malicious content via robots.txt that tells you to slow down to a crawl. Always validate robots.txt against a known-good policy.

A real incident: a misconfigured CMS on a popular domain caused a redirect loop that consumed 40% of a region's crawl budget in under an hour. The fix was to bound redirect depth to 5 hops and add a per-domain redirect limit. Also, monitor per-domain crawl rate and frontier queue depth — a drop in depth with high retry counts suggests a domain is blocking or politeness is too aggressive.

The other lesson: don't let your crawler become the bottleneck for index freshness. Set a maximum crawl age for each domain and stick to it. When the crawler falls behind, the indexer gets hungry and starts serving stale results. That's a user-facing issue that's hard to debug because it's gradual.

DNS caching is another gotcha. A stale DNS entry can cause the crawler to hit a dead IP for hours. Use a TTL cap of 5 minutes and warm multiple DNS resolvers in parallel. If a domain's IP changes faster than your TTL, you'll miss it — compensate by monitoring DNS resolution failures and forcing re-resolution on error.

There's also a subtle politeness trap: some CDNs return a cached 200 even when the origin is down. The crawler thinks the page is fine, but its content is stale. Use a freshness signal from the origin's Last-Modified header. If the header doesn't change, you might be recrawling identical content. Detect content evolution by comparing hashes — if the content hash matches the last fetch, skip reindexing.

One more gotcha: DNS resolution can silently throttle your crawl rate. If your DNS cache TTL is too high, you'll hit dead IPs for hours. Set a cap of 5 minutes and monitor resolution failures. Also, redirect chains are budget killers — always follow a 301 and count hops.

And something that haunted my team: we once had a crawl partition lose its state after a node failure. The frontier coordinator didn't replicate the queue, so we lost 15% of pending URLs. Always persist the frontier state to a durable store or use a replicated log. Recovery time matters less than data loss.

package io.thecodeforge.search.crawler;

import java.util.concurrent.*;
import java.net.URL;

public class CrawlerWorker implements Runnable {
    private final URL frontier;
    private final RateLimiter perDomainLimiter;

    public CrawlerWorker(URL frontier, RateLimiter perDomainLimiter) {\n        this.frontier = frontier;\n        this.perDomainLimiter = perDomainLimiter;\n    }

    @Override
    public void run() {
        while (true) {
            URL url = frontier.next(); // blocking
            perDomainLimiter.acquire(url.getHost());
            // fetch and store...
        }
    }
}

Indexing Pipeline: From Raw Pages to Inverted Index

The indexing pipeline processes crawled HTML through several stages: parsing (extract text, links, metadata), tokenization (split into terms with language-aware segmentation), stop word removal and stemming (e.g., 'running' → 'run'), and finally building the inverted index. For scalability, indexing is done in batches — map-reduce jobs that generate sorted posting lists for each term. Each index segment is an immutable file containing the term dictionary, a frequency table, and the actual posting list. Segments are periodically merged into larger segments to keep the number of files small and lookups fast. The index is distributed across a cluster of index servers, typically sharded by term ID hash.

A detail that bites teams: merging segments is not just a performance concern — it's a consistency one. If a merge fails mid-way, you can lose data from both input segments. Always write merges atomically: write a new segment to a temp path, fsync, then atomically rename it into the index directory. And never skip the fsync — an OS crash during rename can leave you with a half-written segment that looks valid to the reader.

Pain point in production: merge I/O can saturate disk bandwidth. On HDDs, sequential merge I/O can push disk util to 100%, causing query latency to spike by 500ms. Always separate merge and query disks, or use I/O throttling. Use iostat -x 1 to monitor disk await — if >20ms, throttle merge rate. Also, segment merging is write amplification: merging two 10GB segments to a new 20GB segment writes 20GB even if only a small fraction changed. Tiered merging helps: small segments merge frequently, large ones rarely.

Another subtle failure: if a segment's term dictionary becomes corrupted, the entire posting list for a term may be unreadable. Always validate the dictionary checksum after each merge and before exposing the segment to queries.

Compression is not an afterthought. Variable-byte encoding can reduce posting list size by 70%, but decompression adds CPU cost. In production, you balance compression ratio against decode speed — Frame of Reference (FOR) and Delta encoding are common choices. Measure your cache line utilization: posting lists that fit in L1 cache (32KB) are orders of magnitude faster than those that spill into L2.

And here's a real-world production problem: when you add a new segment to the serving tier, you need to warm the OS page cache. If you don't, the first few queries hit a cold cache and latency spikes. Pre-fault the pages by reading the segment headers sequentially before serving.

Also, monitor segment-level health with a background checker that runs a sample query per segment. If any segment returns empty for a known term, flag it immediately. Don't wait for users to report missing results.

Segment corruption is the nightmare that wakes search engineers at 3 AM. Always checksum every segment after a merge and before serving. And don't forget to pre-warm the page cache — a cold segment adds 500ms to the first few thousand queries.

One more nuance: the tokenization stage is often the source of silent recall problems. If you don't handle Unicode normalization, 'Café' and 'cafe' end up as different terms, and you miss matches for queries with accented characters. Use locale-aware tokenization and apply Unicode normalization (NFC/NFD) before indexing.

package io.thecodeforge.search.index;

import java.io.*;
import java.util.*;

/**
 * Simplified segment writer for educational purposes.
 */
public class IndexSegmentWriter {
    private final Map<String, List<Integer>> inverted = new TreeMap<>();

    public void addDocument(int docId, String title, String body) {\n        String text = title + \" \" + body;\n        // In production: tokenize, stem, filter\n        String[] tokens = text.toLowerCase().split(\"\\W+\");\n        for (String term : tokens) {\n            if (term.length() < 2) continue; // skip very short tokens\n            inverted.computeIfAbsent(term, k -> new ArrayList<>()).add(docId);\n        }\n    }\n\n    public void write(String path) throws IOException {\n        try (DataOutputStream out = new DataOutputStream(new BufferedOutputStream(\n                new FileOutputStream(path)))) {\n            out.writeInt(inverted.size());\n            for (var entry : inverted.entrySet()) {\n                out.writeUTF(entry.getKey());\n                out.writeInt(entry.getValue().size());\n                for (int docId : entry.getValue()) {\n                    out.writeInt(docId);\n                }\n            }\n        }\n    }\n}\n

Query Processing and Serving

When a user types a query, the serving layer is responsible for parsing the query, looking up each term in the inverted index, intersecting or merging the posting lists, and ranking the candidate documents — all in under 200ms. The query parser applies spelling correction (e.g., 'gooogle' → 'google'), synonym expansion, and query rewriting based on user intent. Then the term lookup happens across all shards in parallel. Each shard returns its local top-K results (e.g., top 1000), and the serving node merges them into a global top-K. Early termination is critical: you don't need to score all matching documents — you can stop after finding enough high-quality candidates. This is achieved via dynamic pruning algorithms like MaxScore and WAND.

The real pain point in production is the merge step at the root node. If you have 10,000 shards each sending 1000 results, the root has 10 million candidates to rank. That's a lot of work in a few milliseconds. Use a two-phase merge: first, each shard returns only doc IDs and a cheap estimated score; the root filters to, say, 10,000 candidates, then requests full features for those. That cuts merge time by orders of magnitude.

Another hidden issue: the root node's merge is CPU-bound. Profile with a CPU profiler — often sorting a large list of candidate doc IDs is the bottleneck. Replace full sort with a priority queue of size K. Also, if the root node runs out of CPU, query latency degrades linearly across all queries. Pre-compute common query rewrites offline and cache frequent query results aggressively — Google's frontend cache sits in front of the serving layer and handles a large percentage of repeat queries.

Tail latency is the silent killer. Even if 99% of queries complete in 100ms, a single slow shard can push P99 beyond 500ms. Implement hedging: send each query to two replicas and take the first response. Set a timeout per shard step and use early termination to cap worst-case execution time. Monitor root node CPU — if >80% during peak, reduce top-K per shard or add hedging.

Dynamic pruning algorithms deserve a closer look. WAND (Weak AND) maintains an upper bound score for each posting list and skips documents that cannot beat the current K-th best score. MaxScore partitions terms into 'must have' and 'nice to have' groups, only scoring the 'must have' ones early. Both reduce ranking time by 30-50% in practice.

One more nuance: query rewriting can backfire. If you expand a 2-word query into a 10-term OR clause, you blow up the candidate set. Always cap the expansion factor. Also, cache common rewrites: the top 100K queries rarely change their best rewriting. Pre-compute them offline and serve from a lookup table.

The root node merge is deceptively CPU-intensive. If you have 10,000 shards each returning 1000 candidates, that's 10 million documents to rank in milliseconds. Use a two-phase merge: cheap score first, then full features only for the top 10,000. And always set per-shard timeouts.

And here's a failure we hit: a bug in the query rewriter turned every 'the' query into a 50-term OR expansion. The root node got 500 million candidates and melted. The fix: enforce a maximum expansion factor of 10 and a hard cap on candidate count. Also, run a canary query before pushing rewriting changes.

package io.thecodeforge.search.serving;

import java.util.*;

public class QueryParser {
    private final SpellCorrector corrector;

    public QueryParser(SpellCorrector corrector) {
        this.corrector = corrector;
    }

    public ParsedQuery parse(String raw) {
        String corrected = corrector.correct(raw);
        String[] terms = corrected.toLowerCase().split("\\s+");
        return new ParsedQuery(Arrays.asList(terms));
    }

    record ParsedQuery(List<String> terms) {}
}

Ranking and Relevance

Ranking determines which documents appear at the top of the search results page. Early search engines used TF-IDF (term frequency–inverse document frequency), while Google's PageRank used the link graph. Modern search uses multi-stage ranking: first stage (retrieval) uses simple signals like BM25 or vector similarity to retrieve a few hundred candidates; second stage (reranking) applies a more expensive model — often a deep neural network with hundreds of features (e.g., click-through rate, page freshness, user location, query-document match scores). The ranking model is trained offline on user interaction data (clicks, dwell time, skips) and deployed to the online serving stack via model inference servers. Feature computation must be fast — features are precomputed and stored alongside the document in the index.

A subtle issue: features that are computed at query time (like user location) can't be precomputed. That's fine, but be aware that any floating-point operation in the scoring path adds latency. Profile your feature extraction pipeline end-to-end. We've seen teams spend months building a complex DNN only to find that a single feature look-up (e.g., a Redis call for user history) dominates the latency budget. Measure, then model.

Common failure: label leakage during training. If you include future click data as a feature, offline AUC looks amazing (0.99), but online impact is zero. Always validate that all features are available at inference time. Also, model drift is silent — monitor feature distribution drift and set alerting thresholds on model AUC per query cluster. Another real issue: the second-stage reranker can become a bottleneck if it's too slow. Set a timeout per document, and fall back to first-stage scores if the reranker misses its deadline.

One more production headache: feature freshness. If you use a per-document CTR feature that's computed offline, it may be hours old. A page that's trending on social media might have old CTR data, causing the model to underrank it. Set TTLs on features and recompute at least every hour for high-traffic queries.

Another nuance: click modeling is non-trivial. Raw clicks are noisy — a user may click a result by accident. Use dwell time as a quality signal: a click with <2 seconds dwell is likely a bad click. Also, position bias must be corrected: results in position 1 get clicked more regardless of relevance. Use a position bias model to debias training data.

Model monitoring is critical. Track the distribution of each feature in production — if a feature's values drift beyond a threshold, the model is likely stale. Use PSI (Population Stability Index) to quantify drift. Also, monitor the fallback rate: how often does the reranker time out and fall back to first-stage scores? A rising fallback rate indicates the reranker is becoming too slow, perhaps due to increased model complexity or resource contention.

Model drift is silent. You'll see CTR drop over weeks, not hours. Monitor feature distribution drift with Population Stability Index (PSI). And never train on future data — label leakage will give you perfect offline AUC but zero real lift.

One more incident: a team deployed a new ranking model that had a feature 'search_volume_7d' computed from future data. Offline validation looked fine because the feature was computed after the query timestamp. Online CTR dropped 15%. The lesson: always validate feature timestamps in your training pipeline.

package io.thecodeforge.search.ranking;

public class BM25Scorer {
    private final double k1 = 1.2;
    private final double b = 0.75;

    public double score(int termFreq, int docLen, double avgDocLen, int numDocs, int docFreq) {\n        double idf = Math.log(1 + (numDocs - docFreq + 0.5) / (docFreq + 0.5));\n        double tf = (termFreq * (k1 + 1)) / (termFreq + k1 * (1 - b + b * docLen / avgDocLen));\n        return idf * tf;\n    }
}

Distributed Search Serving Architecture

Once the index is built, it must be distributed across a fleet of serving nodes that handle real-time queries. Google uses a two-tier serving architecture: the leaf tier holds index shards, and the root tier aggregates results. Each leaf node hosts a subset of the inverted index sharded by document ID or term. Sharding can be hash-based (e.g., MD5(term) mod N) or range-based. Hash-based gives even distribution but makes prefix queries expensive; range-based helps prefix queries but can cause hotspots. A common compromise is to use hash-based for even load and replicate hot terms across more nodes. Replication is essential for fault tolerance and read throughput — each shard has at least 3 replicas across different racks. Consistency between replicas is eventually consistent: updates are pushed with a version vector, and reads query the most up-to-date replica.

The serving layer also includes a result cache (e.g., for top 10,000 queries) and a frontend that handles query parsing, spelling correction, and personalization before hitting the index servers. Google's frontend cache likes to sit in front of the leafy tier — it can serve up to 80% of repeat queries without touching the index at all.

One gotcha: with shard replication, you must handle replica staleness during index pushes. A common pattern is to use a two-phase commit: prepare new index version on all replicas, then commit. If one replica fails to prepare, abort the entire push to avoid serving inconsistent results.

When adding nodes, consistent hashing avoids full reshuffle, but each node must reload its segment metadata — this can cause a temporary CPU spike. Pre-warm new nodes with routing table before directing traffic. Also, if the routing table is updated asynchronously, some queries may hit a node that no longer holds the segment. Always validate shard ownership on each request via a version check.

Another production issue: version vector conflicts during concurrent pushes. Use a majority read protocol with timestamp ordering to ensure at least one replica returns the latest index version. If versions diverge, the root node should retry on other replicas or fall back to a cached result.

Cross-region replication adds another layer of complexity. Latency between regions can cause significant staleness. Use a primary region for writes and replicate asynchronously. For read affinity, route users to the nearest region and accept eventual consistency. If a region fails, redirect traffic to the nearest healthy region — but be prepared for a spike in latency as caches warm.

Also, think about routing table propagation. In a multi-region setup, the routing table must be synchronized across regions to avoid split-brain scenarios. Use a global distributed consensus like etcd or ZooKeeper to manage routing metadata. Periodically validate that all regions agree on shard ownership.

Consistency during index pushes is the hardest problem. Use a two-phase commit: prepare on all replicas, then commit. If one replica fails, abort the entire push. Never serve from a version vector conflict.

And here's a real outage: a bug in the routing table update caused half the queries to hit a node that had been decommissioned. The fix was to include a generation number in every request and reject stale responses. Also, always run a canary query per shard before and after topology changes.

package io.thecodeforge.search.serving;

import java.util.*;

/**
 * Routes queries to appropriate shards based on term hash.
 */
public class ShardRouter {
    private final List<String> shardNodes;
    private final int replicationFactor;

    public ShardRouter(List<String> shardNodes, int replicationFactor) {\n        this.shardNodes = shardNodes;\n        this.replicationFactor = replicationFactor;\n    }

    public List<String> getNodesForTerm(String term) {
        int hash = Math.abs(term.hashCode());
        int primaryIndex = hash % shardNodes.size();
        List<String> result = new ArrayList<>();
        for (int i = 0; i < replicationFactor; i++) {
            result.add(shardNodes.get((primaryIndex + i) % shardNodes.size()));
        }
        return result;
    }
}

Monitoring and Operational Excellence

A search engine is only as reliable as its weakest stage. You need monitoring at every boundary: crawler health (per-domain crawl rate, frontier depth), indexing pipeline (segment commit latency, merge I/O, corruption checks), serving tier (P50/P95/P99 latency, error rates, cache hit ratio), and ranking (feature drift, model AUC). Without this, you're flying blind.

Alerting thresholds: if crawler per-domain rate drops by 50% in 10 minutes, page. If indexer last commit is older than 2 minutes in a near-real-time index, page. If P99 latency exceeds 200ms for 1 minute, page. If model AUC drops by 0.05 relative to baseline, investigate.

A common failure: silent index corruption without alerting. Always run periodic segment integrity checks (checksum verification) and alert if any segment fails. Also, after each merge, verify the new segment before exposing it.

Capacity planning: index size grows roughly linearly with crawled pages. Monitor storage growth and plan for 1.5x headroom. Shard replicas consume more storage but are necessary for read throughput. Use tiered storage: hot segments on SSD, cold segments on HDD.

Run regular load tests with pre-recorded query logs to validate SLOs under peak. Use chaos engineering to test replica failover and index version rollback procedures. Synthetic queries should be part of your smoke test suite — push a fake query that expects a known result and alert if it fails.

Also, don't forget to monitor the monitoring pipeline itself. If your metrics endpoint goes down, you won't know. Set up a heartbeat check on your monitoring infrastructure and have a separate alerting channel (e.g., PagerDuty) that doesn't rely on the same stack.

Synthetic queries are your best friend. Run a fixed set of queries every minute and alert if results are stale. That catches indexer failures before users do.

One more practice: maintain a runbook for each critical component. When the synthetic query alert fires, you want to know exactly which commands to run. We wrote a runbook that cut mean-time-to-recovery from 45 minutes to 8.

# Crawler health
curl http://crawler-metrics:9090/metrics | grep frontier_depth
curl http://crawler-metrics:9090/metrics | grep per_domain_rate

# Indexer health
curl http://indexer-internal:8080/metrics | grep last_commit_seconds
curl http://indexer-internal:8080/metrics | grep 'segment_count|merge_write_amplification'

# Serving latency (via Prometheus)
query=\"histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))\"
curl \"http://prometheus:9090/api/v1/query?query=$query\"

# Ranking model drift (custom endpoint)
curl http://model-monitor:8080/feature_drift?feature=ctr

Caching and Latency Optimization

Caching is the single biggest lever for reducing search latency. Google employs a multi-tier cache: frontend cache (global CDN for popular queries), segment cache (in-memory posting lists for hot terms), and result cache (entire SERP for frequent queries). The frontend cache alone can serve 80% of repeat queries, reducing latency from 100ms to under 10ms.

But caching brings its own challenges: stale results, cache invalidation, and memory pressure. For the frontend cache, use a TTL of a few minutes for news queries and longer for evergreen content. Use a write-through cache for index segments: when a new segment is published, the old segment's cache entries must be invalidated. A common pattern is to use a generation counter per segment — increment it on every merge, and include it in cache keys.

Another trap: caching too aggressively can mask problems. If you cache a bad result for 5 minutes, that's 5 minutes of user-facing errors. Always cache with a fallback: if a cached result is older than a threshold, refresh it asynchronously and serve the stale version only if the refresh fails.

Memory for cache is finite. Use LRU eviction and monitor cache hit ratios per tier. If the segment cache hit ratio drops below 90%, you're either caching the wrong segments or your working set is too large. Consider adding more nodes or using tiered storage where hot segments stay in memory and cold segments are on SSD.

Cache stampede is a real threat. When a new segment is published, many queries may miss the cache simultaneously and all try to populate it at once. This can spike CPU and overwhelm the index servers. Use a mutex per cache key (distributed lock) to allow only one process to compute the missing value. Others wait and read the newly cached result.

Cache stampede is a real threat. When a new segment is published, many queries miss cache simultaneously and try to populate it. Use a distributed mutex per cache key to prevent duplicate compute.

One more nuance: the result cache should have different TTLs for different query categories. A weather query might be fresh for 15 minutes; a stock price query needs seconds. Use a TTL function based on query type or category. Also, never cache results for queries that have a high rate of change (e.g., 'breaking news') — instead, use a short TTL and a background refresh.

package io.thecodeforge.search.cache;

import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

/**
 * In-memory cache for hot index segments.
 */
public class SegmentCache {
    private final int maxSegments;
    private final Map<String, byte[]> cache = new ConcurrentHashMap<>();
    private final Queue<String> lru = new LinkedList<>();

    public SegmentCache(int maxSegments) {
        this.maxSegments = maxSegments;
    }

    public byte[] get(String segmentId) {
        return cache.get(segmentId);
    }

    public void put(String segmentId, byte[] data) {
        if (cache.size() >= maxSegments) {
            String evicted = lru.poll();
            if (evicted != null) cache.remove(evicted);
        }
        cache.put(segmentId, data);
        lru.add(segmentId);
    }
}

Freshness: Why Yesterday's Index Costs Millions

Users expect Google to know about a news story that broke 5 minutes ago. That means your index must refresh faster than users notice. Most teams treat this as a batch problem — run a nightly job, reindex everything. Wrong move.

The query volume difference between a 6-hour-old index and a 1-minute-old index is measurable in ad revenue. We built a tiered freshness system: critical sources (news, stock prices, social feeds) get a dedicated crawler that re-checks on a 30-second loop. Everything else uses adaptive polling based on historical change frequency. If a page hasn't changed in 72 hours, you don't check it for another hour. If it changes hourly, you check every 5 minutes.

The trick is separating detection from reindexing. A lightweight checksum crawl detects changes without pulling full page content. Only changed pages go through the full indexing pipeline. This cut our index latency by 40x while reducing CPU load by 25%. Freshness is a scheduling problem, not a hardware problem.

// io.thecodeforge — system-design tutorial

import time
import hashlib
from urllib.parse import urlparse

# Exponential backoff based on change history
CHANGE_WINDOW = 5  # track last 5 checks

def should_recrawl(url, change_history):
    if len(change_history) < 3:
        return True
    recent_changes = change_history[-CHANGE_WINDOW:]
    change_rate = sum(recent_changes) / len(recent_changes)
    # High churn = short interval, low churn = long interval
    if change_rate > 0.6:
        return time.time() % 30 < 5  # every 30s
    elif change_rate > 0.2:
        return time.time() % 180 < 5  # every 3min
    else:
        return time.time() % 3600 < 5  # every hour

def check_fresh(url, headers_only=True):
    # Head request yields etag/last-modified, no body
    response = http_head(url)
    return response.headers.get('etag', '')

# Simulating change detection
if __name__ == '__main__':
    history = [0, 1, 1, 0, 1]  # 60% change rate
    print(should_recrawl('http://news.example.com', history))

Spam and SEO Poisoning: The War You Didn't Sign Up For

Google's index is under constant assault from SEO farms, keyword stuffing, link farms, and AI-generated garbage. If your system doesn't actively fight this, your search results become unusable within weeks. The naive approach is to trust the web — bad idea.

We built a three-layer defense. First, pre-index filtering. Before a page even hits the inverted index, we run it through a fingerprint model. If the page text is 95% similar to 50 other pages in the last hour, it's probably spam. We track known spam domains and IP ranges from a shared blocklist updated every 15 minutes.

Second, during ranking, we inject a 'site authority' signal. Pages from domains with high edit-to-page ratios (like Wikipedia) or .edu TLDs get a baseline boost. Pages from domains that have been previously flagged for cloaking or hidden text get a penalty. Third, post-query filtering — if a user searches 'best coffee maker 2024' and 7 of the top 10 results are from the same affiliate network, we deduplicate and drop 5 of them.

The system isn't perfect. You'll never catch everything. But if you don't fight spam explicitly, your index turns into a mirror of the most aggressive marketers, not the most useful content.

// io.thecodeforge — system-design tutorial

from collections import Counter
import re

# Simple fingerprint: extract top 10 content words
def content_fingerprint(text):
    words = re.findall(r'\w+', text.lower())
    common = Counter(words).most_common(10)
    return set(w for w, _ in common)

SPAM_THRESHOLD = 0.95  # 95% fingerprint overlap = spam
seen_fingerprints = []

def is_spam(html_text, domain):
    blocklist = ['spamfarm.example.com', 'seohell.net']
    if domain in blocklist:
        return True
    fp = content_fingerprint(html_text)
    for existing in seen_fingerprints[-200:]:  # sliding window
        intersection = fp.intersection(existing)
        union = fp.union(existing)
        if len(union) > 0 and (len(intersection) / len(union)) > SPAM_THRESHOLD:
            return True
    seen_fingerprints.append(fp)
    return False

# Test
if __name__ == '__main__':
    page1 = "buy cheap gadgets now best gadgets cheap price"
    page2 = "cheap gadgets buy now! best price for gadgets"
    print(f"Page1 is spam: {is_spam(page1, 'goodsite.com')}")
    print(f"Page2 is spam: {is_spam(page2, 'goodsite.com')}")