Full-Text Search

Full-text search lets users query large volumes of unstructured text and get results ranked by relevance, not just exact matches. It works by building inverted indexes that map every word to the documents containing it, enabling sub-second lookups across millions of records. Technologies like Elasticsearch, Apache Solr, and PostgreSQL's tsvector power search features in virtually every modern application, from e-commerce product search to log analysis.

Why This Matters

Full-text search is fundamental because nearly every user-facing application requires some form of search. A naive LIKE '%term%' SQL query scans every row and cannot rank results by relevance — it's unusable at scale. Full-text search engines solve this with inverted indexes that turn the problem inside out: instead of scanning documents for words, they look up words to find documents. Understanding how tokenization, stemming, scoring, and index management work is critical for designing systems that return the right results fast while keeping indexes fresh and storage manageable.

The Building Blocks

• Inverted Index: The core data structure that maps each unique term to a list of document IDs containing it, enabling O(1) term lookups instead of full table scans.
• Tokenization & Analysis: Text is split into tokens, lowercased, stripped of stop words, and stemmed (e.g., 'running' → 'run') to normalize matching across different word forms.
• Relevance Scoring: Algorithms like BM25 and TF-IDF rank documents by how frequently and uniquely a term appears — a word in 3 documents scores higher than one in 3 million.
• Analyzers & Mappings: Custom analyzers define how text is processed — language-specific stemming, synonym expansion, n-gram tokenization for autocomplete, or phonetic matching.
• Sharding & Replication: Search indexes are sharded across nodes for parallelism and replicated for fault tolerance, allowing horizontal scaling of both indexing and query throughput.

Under the Hood

When a document is indexed, the analysis pipeline first tokenizes the raw text into individual terms. Each term passes through a chain of filters — lowercasing, stop word removal, stemming or lemmatization — producing normalized tokens. These tokens are stored in an inverted index: a dictionary where each term points to a posting list containing document IDs, term frequencies, and positional data.

At query time, the same analysis pipeline processes the search query, producing normalized tokens that are looked up in the inverted index. The search engine retrieves the posting lists for each query term, intersects or unions them based on boolean logic, and scores each matching document. BM25, the standard scoring algorithm, considers term frequency (how often the term appears in the document), inverse document frequency (how rare the term is across all documents), and field length normalization.

For scale, the index is divided into shards distributed across nodes. Each shard processes the query independently and returns its top results. A coordinating node merges these partial results into a globally ranked list. Techniques like segment merging, doc values for sorting, and filter caching ensure queries stay fast even as the index grows to billions of documents.

How Companies Actually Do This

Elasticsearch at GitHub Powers code search across 200+ million repositories, using custom analyzers for code tokenization (splitting on camelCase, dots, underscores) and sharding indexes by repository to parallelize queries.

Amazon Product search on Amazon uses a full-text search engine with synonym expansion, spell correction, and personalized ranking to handle millions of queries per second across a catalog of hundreds of millions of products.

Slack Uses Elasticsearch to index billions of messages across millions of workspaces, enabling instant search with access control filters that ensure users only see messages from channels they belong to.

Common Pitfalls

1. Not keeping search indexes in sync with the source database — stale indexes cause users to find deleted items or miss new ones; use change data capture or transactional outbox patterns
2. Over-indexing fields that don't need full-text search — indexing numeric IDs or timestamps as analyzed text wastes storage and slows indexing without improving search quality
3. Ignoring relevance tuning — default BM25 scoring rarely matches business needs out of the box; you need field boosting, function scoring, and user feedback loops to return useful results

Interview Questions Worth Practicing

1. How would you design a search system that stays in sync with a relational database containing millions of products?
2. Explain how an inverted index works and why it's faster than SQL LIKE queries for text search.
3. How would you handle multi-language search in a system serving users in 20+ countries?

The Tradeoffs

• Storage vs Speed: Inverted indexes duplicate data from the source, consuming significant disk space, but enable sub-second queries across millions of documents
• Freshness vs Throughput: Real-time indexing keeps results current but reduces write throughput; batch indexing is faster but introduces a delay before new content is searchable
• Precision vs Recall: Strict matching returns fewer but more accurate results; stemming and synonym expansion return more results but may include less relevant ones

How to Explain This in an Interview

Here is how I would explain Full-Text Search in a system design interview:

In an interview, start by explaining the inverted index — it's a map from every unique word to the list of documents containing that word, which flips the search problem from scanning all documents to a fast dictionary lookup. Then explain the analysis pipeline: text is tokenized, lowercased, stemmed, and stripped of stop words before indexing. At query time, the same pipeline runs on the user's query so terms match consistently. Results are ranked using BM25, which scores documents higher when they contain rare terms frequently. For scale, mention sharding the index across nodes so queries run in parallel. Finish by noting the key operational challenge: keeping the search index in sync with the source of truth, typically solved with change data capture or an event-driven pipeline.

Related Topics

• Database Indexes
• Database Types
• Caching Strategies
• NoSQL Data Modeling

The Real-World Incident That Made This Famous

Understanding Full-Text Search became critical after multiple high-profile production incidents at major tech companies. When systems handle millions of users, even small misunderstandings about Full-Text Search can lead to cascading failures that cost millions in lost revenue and erode user trust. Companies like Netflix, Google, Amazon, and Meta have all invested heavily in mastering Full-Text Search because they learned the hard way that ignoring it leads to outages.

The key lesson from these incidents: Full-Text Search is not just a theoretical concept — it is a practical skill that separates engineers who build resilient systems from those who build fragile ones. Every major outage report from the past decade involves at least one Full-Text Search-related design decision that was either implemented incorrectly or overlooked entirely during the initial architecture review.

How Senior Engineers Think About This

Senior engineers approach Full-Text Search differently from textbook definitions. Instead of memorizing rules, they build mental models. They ask: "What problem does Full-Text Search solve? When does it fail? What are the alternatives?" This problem-first thinking leads to better design decisions because every system has unique constraints.

When evaluating Full-Text Search in a system design context, experienced engineers consider the failure modes first. What happens when this component goes down? How does the system degrade? Is the degradation graceful or catastrophic? These questions reveal more about your understanding than any textbook definition.

The key difference between junior and senior engineers when it comes to Full-Text Search: juniors focus on the happy path, while seniors design for what happens when things go wrong. They consider operational cost, team expertise, monitoring requirements, and how the decision will look six months from now when traffic has grown 10x.

Common Interview Mistakes

Mistake 1: Giving a textbook definition without context. Interviewers want to see you connect Full-Text Search to real systems and real problems. Instead of reciting definitions, explain when and why you would use Full-Text Search in the system you are designing.

Mistake 2: Not discussing trade-offs. Every design decision involving Full-Text Search has trade-offs. Discuss what you gain and what you give up. Acknowledge the downsides and explain why the benefits outweigh them for your specific use case.

Mistake 3: Overcomplicating the solution. Start with the simplest approach to Full-Text Search that meets the requirements, then add complexity only when justified. Many candidates jump to complex implementations when a simpler solution would work perfectly.

Production Checklist

• Define clear metrics for measuring the effectiveness of your Full-Text Search implementation
• Set up monitoring and alerting that specifically tracks Full-Text Search-related failures
• Document your Full-Text Search design decisions in Architecture Decision Records (ADRs)
• Test failure scenarios related to Full-Text Search in staging before production deployment
• Review and update your Full-Text Search implementation quarterly as system requirements evolve
• Train new team members on the specific Full-Text Search patterns used in your system
• Establish runbooks for common Full-Text Search-related incidents and recovery procedures

Practical Implementation for .NET Developers

In .NET, PostgreSQL full-text search works via Npgsql and EF Core — map tsvector columns and query with EF.Functions.ToTsVector and ToTsQuery for GIN-indexed searches. For dedicated search, the NEST and Elastic.Clients.Elasticsearch NuGet packages provide strongly-typed clients for Elasticsearch. You can also use Azure.Search.Documents for Azure Cognitive Search, which offers built-in AI enrichment. For simpler scenarios, Lucene.NET provides an embeddable full-text search library without external dependencies.

ASP.NET Core setup: Create a service class that encapsulates the logic, register it with dependency injection, and inject it into your controllers or minimal API endpoints. The built-in DI container handles lifecycle management.

Entity Framework Core: For database interactions, EF Core provides the ORM layer. Use migrations for schema management and raw SQL for performance-critical queries. Consider Dapper for read-heavy paths where EF Core overhead matters.

Azure integration: If deploying to Azure, leverage managed services — Azure Cache for Redis, Azure SQL, Azure Service Bus, Azure Cosmos DB. These eliminate operational overhead and provide built-in monitoring through Application Insights.

Testing: Use xUnit with Testcontainers for integration tests that spin up real databases in Docker. Mock external dependencies with NSubstitute. The WebApplicationFactory class lets you test your entire HTTP pipeline in-process.

Monitoring: Add Application Insights telemetry to track request latency, dependency calls, and custom metrics. Use structured logging with Serilog to make production debugging possible:

Log.Information("Processing {Operation} for {ResourceId}", operation, resourceId);

This gives you searchable, structured logs in Azure Monitor or Seq.