How AI Search Works: From Keywords to Vector, Hybrid Search and RAG

Over the years, search has become the primary way we interact with the internet, our computers, and other smart devices.

The verb “to google” is now considered synonymous with looking things up online, and some university professors have even observed that their students increasingly fail to grasp the concept of organizing files into directories because search is everywhere. This shows the central role search plays in our digital experience today.

Decades of research in information retrieval, the science of finding information, have fueled advances in search that now form the backbone of a wide variety of applications. Recently, however, new technologies have entered the scene. Advances in artificial intelligence have taken the tech world by storm, as shown by the popularity of large language models (LLMs) such as OpenAI’s ChatGPT, Anthropic’s Claude, Google’s Gemini, Meta’s Llama, and many more. These models, along with a series of significant breakthroughs in machine learning, have opened a whole new frontier in search, making it possible to handle complex queries and questions in ways that were previously impossible.

In this guide, we’ll survey some of the most important recent advances in search technology. After a brief introduction to traditional search methods, we’ll explore techniques that use machine learning and natural language processing (NLP), focusing on vector and hybrid search. Then we’ll look at how artificial intelligence can be integrated into search, and how approaches like retrieval-augmented generation are shaping its future.

The basics of search

Our focus throughout this guide is primarily on text search: the task of finding relevant written documents for a user’s query. Search results can be considered relevant if they align well with the query and its intent.

For example, if a user is interested in visiting a cozy “Christmas market,” relevant results for this term might include pages titled “Top 10 Christmas Markets in Europe,” “Local Christmas Market Events Near You,” or “Guide to the Best Christmas Markets This Holiday Season.” In contrast, pages titled “Stock Market Trends in December,” “Christmas Tree Farming Techniques,” or “The Best Supermarkets for Holiday Shopping” would not be relevant.

Full-text search

Many traditional search engines rely on full-text search, a family of algorithms that analyze the input and candidate texts to provide related results. The exact details vary from one algorithm to another, but full-text search solutions generally consist of two main steps:

1. Indexing: analyzing documents to create a searchable index, often an inverted index. This results in a data structure that maps terms to the documents containing them, which enables efficient retrieval.
2. Retrieval: identifying the most relevant documents by applying a ranking function. The results are presented to the user ranked by their relevance to the query.

Depending on the scale at which a search engine operates, an index may be relatively small (for example, all the pages on a small company’s website) or massive, up to hundreds of billions of webpages (for example, Google’s search index).

Text processing for full-text search

Full-text search involves a number of text processing steps, which allow search engines to do more than simple keyword matching. For instance, when indexing data, modern search systems generally perform stemming. This is the process of reducing words to their root form, to lessen the influence of grammatical inflections on the results. The word “walking,” for example, can be reduced to the root “walk” and the suffix “-ing.” But the word can take other forms as well, such as “walks,” “walked,” and “walker.” Stemming lets search engines recognize that these are related and treat them that way.

Another very common text processing step is stop word removal. Common but less meaningful words, such as articles, prepositions, and other grammatical particles, are filtered out. Examples include “a,” “an,” “the,” “is,” “are,” “was,” “were,” “in,” “on,” and “at.” Since these words occur very frequently across languages, they can lower overall relevance, and are usually not taken into account when searching.

Further text processing methods include tokenization (breaking sentences down into individual words), normalization (for example, lowercasing everything), and lemmatization (similar to stemming, but reducing words to their dictionary forms, which keeps more meaning). These methods all serve the same goal: converting the contents of a document into a form that is easier for search engines to work with.

Retrieval

Once the documents have been processed and indexed, the engine is ready to perform retrieval. This is where search begins, making it the part of a search system most users are familiar with.

To retrieve relevant information, full-text search engines typically use a ranking function. These functions assign each document in the index a score that represents how well it matches the query. One of the most widely used is the BM25 algorithm (BM stands for “best matching”). BM25 builds on term-frequency methods, which consider how often query terms appear both within each document and across the entire collection to determine how informative they are. The algorithm also uses a set of tunable parameters and weighting strategies to improve its rankings, making it one of the best-performing scoring methods available today.

Thanks to the good balance it strikes between performance and simplicity, BM25 remains one of the most popular ranking algorithms, despite having been an industry standard for decades. It’s also the default scoring method in many search applications, including popular open-source software such as Apache Lucene and OpenSearch.

Towards intelligent search

Full-text search may be a powerful tool in the search engineer’s toolbox, but it is most useful when results based on exact or near-exact matches are enough. In other words, it might be perfect for search applications where users already have a fairly clear idea of what they’re looking for, and can use keywords to find it. But in applications that need to support a more open-ended kind of interaction with content, traditional search engines often fall short.

Challenges in traditional search

Human language is a complicated system, where words can be ambiguous and the same concept can be described in different ways. People phrase the same idea in various ways, and when searching, they often don’t have a clear idea of exactly how to describe what they’re looking for. What’s more, the language we use with each other doesn’t necessarily align well with how language is represented in computer systems.

In search, the complexity of language creates a number of challenges. For example, traditional full-text search doesn’t handle synonyms or misspellings very well. Instead, these have to be explicitly coded to produce meaningful results. If a person is looking for information about getting a “construction permit” on a municipality’s website, but the site only contains articles about getting a “building license,” they might not find the answers they need.

Traditional search may also not provide broad enough context. The same person would probably benefit from learning about construction rules in general, but may not be shown that content if the results rely too heavily on the keywords “construction” and “permit”.

Semantic search

To address these challenges, a new approach known as semantic search has emerged. This family of retrieval methods offers an alternative to matching keywords. Semantic search aims to understand the intent and contextual meaning behind search queries, so it can deliver results that are more helpful and relevant.

In practice, semantic search is an umbrella term for a combination of approaches that make traditional search more context-aware. It typically involves the following techniques:

• Query analysis: Using advanced NLP techniques, semantic search engines can better understand the input. For example, Named Entity Recognition can identify proper nouns, such as the names of places, people, and organizations.
• Query expansion: NLP can also enhance queries by adding synonyms and related concepts, broadening the coverage of the results. The query “driver’s license,” for instance, can be expanded to “driver’s license registration, application form, and renewal.”
• Knowledge graphs: These encode relationships between entities, which can improve results for queries that need factual knowledge. Relationships are often represented as triples, such as “:JamesCameron :directed :Titanic”.
• Personalization: Beyond understanding relationships between entities in text, semantic search systems can also use contextual information based on user behavior, such as history, location, or search preferences. Recommendations based on this kind of data make results more relevant to each user’s needs.

Despite the considerable benefits semantic search brings to both relevance and user experience, it often requires carefully engineered, domain-specific implementation. Developers must make sure, for example, that queries are analyzed and enhanced in ways that are relevant to the contents of the index. Synonym dictionaries need to be built with this in mind. A word like “bank,” for instance, means something very different in finance than in geography, where it may mean a riverbank rather than a financial institution. Similarly, knowledge graphs must be built for a specific domain to accurately represent its entities and relationships.

AI-powered search

Recognizing the limits of technologies that require manually creating resources to represent linguistic and behavioral patterns, search engineers have turned to machine learning. This has led to a growing focus on algorithms that can learn these representations themselves.

Machine learning (ML), a branch of artificial intelligence (AI), aims to enable computers to learn from vast amounts of data. ML algorithms are built to make decisions and perform tasks on their own, without human supervision. Advances in machine learning have transformed industries from entertainment to medicine. Content recommendations on streaming platforms rely on ML models, much as disease diagnosis and drug discovery rely on algorithms trained on extensive datasets. And of course, ML has been a key driver of the ongoing AI revolution.

Machine learning has also had a transformative impact on information retrieval. Search engines today increasingly use advanced ML techniques to provide more relevant and actionable results. Some of the most influential include text embeddings, vector and hybrid search, and retrieval-augmented generation.

Vector search

Representing natural language in a way computers can understand has long been a challenge. However, a technique known as text embeddings, vector representations of words and sentences, has brought us a step closer to a solution. Grounded in distributional semantics, the idea that words and phrases can be compared to each other based on the contexts they appear in, these numerical representations make it possible to measure how similar two pieces of text are in meaning.

Word vectors, for instance, can tell us that the words “France” and “Paris” relate to each other in the same way as “Germany” and “Berlin.” Vectors that encode longer chunks of text can show these relationships too. For example, they can tell that the following sentences are very similar in meaning:

• The capital of Denmark is a bike-friendly city blending history and modern design.
• Copenhagen boasts excellent cycling infrastructure and architecture that blends tradition with contemporary elements.

Simplified illustration. Real embeddings have hundreds or thousands of dimensions.

Vectors are learned from large amounts of data using machine learning. The resulting model is essentially a program that converts text into a string of numbers, which encodes each vector’s position and relationship to other vectors in a vector space. These vectors can then measure similarity across many dimensions of meaning, since words and phrases that are close in meaning tend to be neighbors in the vector space. So we can compare the vector of a search query to those of the documents in an index, and find the ones that are most relevant in meaning.

This process, known as nearest-neighbor search, brings several improvements over the strictly text-based methods we discussed earlier. For instance, vector search handles synonyms and variations in word choice much better. It’s also resilient to typos and misspellings, which usually need to be corrected in full-text search systems. Significant as these advantages are, the main strength of vector search is that it gives a better overall understanding of the query and its intent, which works especially well for longer, natural-language searches.

Hybrid search

Despite the fundamental differences in how they work, full-text search and vector search shouldn’t be seen as competing approaches. In fact, combining them can pave the way toward a truly intelligent and robust search solution. On one hand, keyword matching is lightweight, easy to implement, and works extremely well where exact matches and predictable outcomes are needed. On the other, vector search shines in applications that need a deeper understanding of meaning. Hybrid search, a combination of the two, represents the best of both worlds.

Hybrid search can be implemented in different ways, but the most common approach is to run queries against both a full-text and a vector search engine, then combine the results using a fusion algorithm. Typically, the goal is to merge the results so that both the keyword and vector similarity scores contribute to the final ranking, or at the very least, so the strengths of both approaches are used.

One of the most popular fusion algorithms, reciprocal rank fusion (RRF), was developed with exactly this purpose in mind. RRF merges documents from several result sets by considering how highly they rank in each. The final ranking comes from adding up the reciprocal of each document’s rank in each result set, then sorting accordingly. The algorithm doesn’t need manual tuning, and works even when the scores in the original rankings are on different scales. It’s simple and computationally efficient, which explains its popularity in modern hybrid search.

Today, there is a clear trend toward hybrid search becoming the standard across the search industry. Engineers and providers alike understand the importance of giving search semantic capabilities while keeping the strengths of traditional methods. Hybrid search is the most effective way to do this, which is why demand for it is growing rapidly.

Beyond search: retrieval-augmented generation

Search may well be the standard way to look for information online, but a list of blue links isn’t always the best way to present it. In the era of AI assistants and conversational search, integrating a large language model into a search system can substantially improve the overall user experience.

LLMs are a powerful technology, but they can generate incorrect or misleading information. Known as hallucinations, these errors can seriously undermine people’s trust in the systems that use these models. Solving hallucinations is an ongoing area of research, but a technique called retrieval-augmented generation (RAG) can often reduce the problem successfully.

In essence, RAG combines the accuracy and reliability of search with the conversational fluency of generative AI models. At its core, it relies on a fairly simple idea: given a search query and a set of relevant documents, the language model generates a helpful response.

One area where this approach is especially useful is chatbots on content-heavy websites. In the past, chatbots often relied on hand-crafted rules and decision trees. RAG can simplify this immensely. With a robust semantic search engine connected to an LLM, chatbots can answer all sorts of questions about a website’s content in a natural-sounding way. RAG is also useful for generating summaries of the search results that are most relevant to a query.

Perhaps most importantly, retrieval-augmented generation limits what information LLMs can include in their responses. This is why the approach is widely considered one of the most effective ways to combat hallucinations today. Organizations need to make sure that if they use an LLM in customer-facing applications, its responses are factually accurate and up to date. Thanks to the underlying search system, RAG can make all the difference between fabricated, and therefore inaccurate, information and reliable answers that come from a verified source.

Conclusion

Through decades of research and development in information retrieval, search has matured from simple keyword matching into a diverse range of intelligent solutions. Technologies like hybrid search and retrieval-augmented generation have shown not only that it’s possible to combine natural language processing with the latest advances in artificial intelligence to build smarter search, but also that this can be done without throwing away traditional, efficient methods. Today, AI is driving change in search at an unprecedented rate. Ultimately, the future is in our hands, and there is much to look forward to.

Written by András Aponyi, Cludo