Full-Text Search & Elasticsearch: A Mental Model for Backend Engineers
If you've ever written WHERE name ILIKE '%laptop%' in production, you've already met the problem this article solves. You just haven't felt it yet.
This is part 11 of my backend-from-first-principles series. We've covered how requests enter your system, how data is stored, how it's cached. This one is about a specific question that breaks every backend the moment scale shows up: given a few words from a user, how do you find the most relevant rows in a table of millions — in under a second?
The answer is not "a faster database." The answer is a fundamentally different data structure.
The Setup: It's 2005, and Your Search Works Fine
Picture yourself as a backend engineer at a growing e-commerce company in 2005. You have 5,000 products. The product manager asks for a search endpoint. You write this:
SELECT * FROM products WHERE name ILIKE '%laptop%' OR description ILIKE '%laptop%';
It works. Latency is ~50ms. Users are happy. You move on.
Two years later, the catalog has 5 million products. The same query now takes 30+ seconds. Worse, when a user searches for "laptop", the top result is a laptop bag instead of a MacBook Pro. The query is slow and wrong.
Two things broke at once:
- Speed — the database is reading every row, top to bottom, scanning every character of every name and description. There is no shortcut.
- Relevance — even when it returns results, the database has no concept of "more relevant." It returns rows in whatever order it found them.
Both problems trace back to a single root cause: the database is searching inside documents, when it should be searching for words that point to documents.
The Mental Model: The Index at the Back of a Textbook
Open any textbook. Flip to the back. There's an index that looks like this:
inheritance ......... 42, 87, 156 inversion ........... 203 inverted index ...... 198, 199, 211
If I asked you "where does the book talk about inverted indexes?", you wouldn't read all 400 pages cover to cover. You'd flip to the back, find the word, and jump straight to pages 198, 199, 211.
That's it. That's the entire idea behind full-text search.
A normal database is the textbook with no index — every search means reading the whole book. A full-text search engine is the textbook with the index — every search is a single lookup followed by a jump.
The flipped relationship is the key:
- Document → Words (how data is stored): Page 198 contains the words "inverted", "index", "search", "lookup"...
- Word → Documents (how the index is structured): The word "inverted" points to pages 198, 199, 211.
Reverse the arrow, and search becomes O(1). That reversed mapping has a name: the inverted index.
The Librarian vs. The Index Card Catalog
Another way to feel the difference:
The relational database is a librarian without a catalog. You walk up and say "I want books about machine learning." The librarian sighs, walks to shelf one, opens book one, scans it for "machine learning", closes it, walks to shelf two, opens book two, scans it... for every book in the library. Comprehensive. Slow. Useless at scale.
A search engine is the same library with a card catalog. You walk up, the librarian flips to the "M" drawer, pulls out the card for "machine", and the card already lists every book that contains that word, with the page numbers. One lookup. Done.
The card catalog wasn't free. Someone built it once, painstakingly, and updates it every time a new book arrives. That's the tradeoff: full-text search trades write-time work for read-time speed. You spend more effort indexing data when it's added, so that searches become near-instant.
How an Inverted Index is Built
Suppose you have three product descriptions:
doc_1: "Apple MacBook Pro laptop" doc_2: "Dell laptop charger" doc_3: "Apple iPhone case"
The search engine doesn't store these as raw strings. It runs them through a pipeline called analysis:
- Tokenization — split text into individual words (tokens).
"Apple MacBook Pro laptop"→["Apple", "MacBook", "Pro", "laptop"]. - Lowercasing — collapse case so
"Apple"and"apple"match. →["apple", "macbook", "pro", "laptop"]. - Stop-word removal — drop common, low-information words like "the", "a", "and", "of". They appear in almost every document and tell you nothing about relevance.
- Stemming / lemmatization — collapse word variants to their root form.
"running","runs","ran"→"run". So a search for one matches all three.
After analysis, the engine builds the inverted index:
apple → [doc_1, doc_3] macbook → [doc_1] pro → [doc_1] laptop → [doc_1, doc_2] dell → [doc_2] charger → [doc_2] iphone → [doc_3] case → [doc_3]
Now a search for "laptop" isn't a scan. It's a single dictionary lookup that returns [doc_1, doc_2] instantly — no matter whether the catalog has 5,000 products or 500 million.
The inverted index is to text what a B-tree index is to numeric IDs. Both transform a linear scan into a logarithmic (or constant-time) lookup. The only difference is what gets indexed: words instead of column values.
The Second Half of the Problem: Relevance
Speed is solved. But the user searches for "laptop" and the index returns 4,200 documents that contain the word "laptop". In what order do you show them?
This is where SQL gives up entirely. SQL knows which rows match. It has no opinion about which match is better. A search engine does — through a scoring algorithm called BM25.
BM25 is the standard ranking function used by Elasticsearch, Lucene, and most modern search systems. You don't need to memorize the formula. You need to understand the four signals it combines:
- Term Frequency (TF) — How often does the search term appear in this document? A document mentioning "laptop" 8 times is probably more about laptops than one mentioning it once.
- Inverse Document Frequency (IDF) — How rare is this term across the entire corpus? "The" appears in every document, so it carries no signal. "MacBook" appears in only 2% of documents, so a match on "MacBook" is much more meaningful than a match on "the".
- Document Length — Shorter documents win ties. A 5-word product title containing "laptop" is more likely to be about laptops than a 5,000-word review that happens to mention "laptop" once.
- Field Boosting — A match in the
titlefield is worth more than a match in thedescriptionfield. You configure these weights when defining the index.
Combine these and you get a ranked score per document. Sort by score descending, and the user sees MacBook Pro first, laptop bag tenth — instead of whatever insertion order the database happened to return.
Speed comes from the inverted index. Relevance comes from BM25. You need both. An engine without scoring is just a fast match — it isn't search.
Enter Elasticsearch
You don't build inverted indexes and BM25 scorers yourself. You use a search engine — and in 99% of backend codebases, that engine is Elasticsearch.
Elasticsearch is a distributed wrapper around Apache Lucene, the original Java library that implements inverted indexes and BM25. Lucene is the engine; Elasticsearch is the car around it — adding clustering, REST APIs, a JSON query language, and operational tooling.
What Elasticsearch gives you out of the box:
- Inverted index, automatically. You ingest a JSON document; Elasticsearch tokenizes, normalizes, and indexes every text field. No setup beyond a simple mapping.
- Relevance scoring. BM25 by default. Configurable weights, custom scoring scripts if you need them.
- Fuzzy matching. A user types
"treading"; Elasticsearch matches"trending"via edit-distance algorithms (Levenshtein). Typos forgiven. - Autocomplete and prefix search. Power "search-as-you-type" interfaces with sub-50ms responses.
- Horizontal scaling. Indexes are split into shards, distributed across nodes. Add hardware, capacity grows linearly.
- Aggregations. Beyond search — group, count, average, histogram. Same engine that finds documents can also produce analytics dashboards in real time.
The Benchmark: Postgres vs. Elasticsearch on 50,000 Reviews
Here's a concrete comparison. Same dataset — 50,000 product reviews — loaded into both systems.
Postgres setup:
CREATE TABLE reviews ( id SERIAL, review TEXT, sentiment TEXT ); -- search query: SELECT * FROM reviews WHERE review ILIKE '%searchTerm%';
Elasticsearch setup:
{
"mappings": {
"properties": {
"review": { "type": "text" },
"sentiment": { "type": "keyword" }
}
}
}
Results:
| Search Term | Elasticsearch | Postgres ILIKE |
|---|---|---|
laptop | ~1.0 sec | ~3.0–4.0 sec |
something | ~0.5 sec | ~7.0 sec |
Both systems return identical match counts. The difference isn't correctness; it's the work being done. Postgres scans every row and pattern-matches every string. Elasticsearch does a hash lookup in the inverted index and pulls out a pre-built list of document IDs.
And remember — this is at 50,000 rows. Extrapolate to 50 million and the gap becomes the difference between "instant" and "your service is down."
Postgres Full-Text Search: The Honest Middle Ground
Before you provision an Elasticsearch cluster, know that Postgres has its own full-text search built in. It's not as powerful as Elasticsearch, but it's significantly faster than ILIKE.
-- Add a tsvector column for full-text indexing
ALTER TABLE products
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (
to_tsvector('english', name || ' ' || description)
) STORED;
CREATE INDEX idx_search ON products USING GIN(search_vector);
-- Query:
SELECT * FROM products
WHERE search_vector @@ plainto_tsquery('english', 'laptop')
ORDER BY ts_rank(search_vector, plainto_tsquery('english', 'laptop')) DESC;
Under the hood, Postgres builds an inverted index using a GIN (Generalized Inverted Index). It does its own tokenization and stemming. It supports ts_rank for relevance scoring (a simpler cousin of BM25). It handles tens of millions of rows comfortably.
The honest framing:
- Postgres FTS is great when search is a feature of your app, not the core product. You already have Postgres. You don't want another service to operate.
- Elasticsearch is great when search is the product, or when you need fuzzy matching, faceted filtering, geo search, real-time analytics, or scale beyond what one Postgres node can handle.
Don't reach for Elasticsearch on day one. Most products start fine on Postgres FTS and only outgrow it when search becomes a primary differentiator. Premature distributed-systems complexity has killed more projects than slow search ever did.
When to Use What
| Scenario | Use |
|---|---|
| Small dataset, exact-ish match, simple LIKE works | Plain SQL |
| Medium dataset, real text search, you already run Postgres | Postgres FTS (tsvector + GIN) |
| Large dataset, advanced relevance, fuzzy matching, autocomplete | Elasticsearch |
| Log analysis and observability at scale | Elasticsearch (ELK stack) |
| Real-time analytics on text-heavy data | Elasticsearch (aggregations) |
Beyond Search: The ELK Stack
Once you understand Elasticsearch, you understand why every observability product on Earth uses it. Logs are just text documents at scale — billions of them, written every second. The same inverted-index machinery that makes product search fast also makes log search fast.
That's the ELK stack:
- E — Elasticsearch. Stores and indexes the logs.
- L — Logstash. A pipeline that ingests logs from anywhere (files, syslogs, queues), parses them, normalizes them, and ships them to Elasticsearch. Modern setups often use Fluentd or Beats instead.
- K — Kibana. The visualization layer. Search the logs, build dashboards, set up alerts. The thing your SRE team stares at during incidents.
When a production bug hits, you grep across petabytes of logs in under a second using a Kibana query. That's the inverted index at work — same algorithm, different domain.
The Tradeoffs You Need to Know
Elasticsearch isn't free magic. It comes with real costs:
- Eventual consistency. When you write a document, it's not immediately searchable. Elasticsearch refreshes its in-memory index every ~1 second by default. For most search use cases, that's invisible. For "user updates their profile and immediately searches for it", it can surprise you.
- Not the source of truth. Elasticsearch is a search index, not a database. Treat your relational DB as truth and Elasticsearch as a derived view. If the cluster dies, you should be able to rebuild the entire index from your primary store.
- Operational overhead. Clusters, shards, replicas, JVM tuning, mapping migrations. It's a real piece of infrastructure to operate. Most teams reach for managed offerings (Elastic Cloud, OpenSearch on AWS) for this reason.
- Index = storage cost. The inverted index is a copy of your data, structured differently. You're paying for two stores: your primary database and your search index.
The Mental Model Summary
Carry this forward:
- The flipped arrow = Documents-to-words is how data is stored. Words-to-documents is how the inverted index is structured. That single reversal is the entire trick.
- Inverted index = The back-of-the-textbook index. One lookup, jump straight to results. Built once at write time, queried instantly at read time.
- Analysis pipeline = Tokenize → lowercase → strip stop-words → stem. Same pipeline runs on indexed text and on incoming queries — that's why "Running" matches "ran".
- BM25 = Term frequency + inverse document frequency + length normalization + field boosting. The four signals that turn matches into rankings.
- Lucene → Elasticsearch = Lucene is the engine, Elasticsearch is the distributed JSON-API wrapper around it. OpenSearch is AWS's fork.
- Postgres FTS = Real full-text search inside Postgres via tsvector + GIN. Use it before Elasticsearch unless you actually need fuzzy matching, scale, or analytics.
- Search index ≠ database = Elasticsearch is a derived view. Your primary store is still the source of truth. You should be able to rebuild the index from scratch.
- ELK stack = Same algorithm, different problem. Logs are text. Search infrastructure is also log infrastructure.
- The rule = SQL
LIKEreads every row. Inverted indexes read no rows — they read a dictionary. That's the entire performance gap.
The inverted index is one of those quiet ideas that sits underneath half of modern software — every product search, every Slack message lookup, every Kibana query, every "search your inbox" feature. Once you see the back-of-the-book pattern, you start spotting it everywhere. And the day you understand why WHERE name ILIKE '%term%' doesn't scale is the day search stops being magic and becomes engineering.
This is part 11 of my "Backend from First Principles" series. Everything in this article comes from what I learned watching the Backend from First Principles playlist by Sriniously — specifically the video on full-text search and Elasticsearch. I'm not claiming original research. This is me taking what I studied, internalizing it, and presenting it as a mental model that made the concepts stick for me. If it helps you too, even better.
Follow along for the next one.