What is RAG? How businesses get AI to answer from their own files

Ask a general-purpose AI chatbot about your company’s refund policy and it will guess. Retrieval-augmented generation (RAG) is the fix most businesses reach for. Before the model replies, the system searches your own files for the passages that best match the question and pastes them into the prompt, so the answer rests on your current data rather than only on what the model picked up in training.

Key takeaways

RAG searches your documents first and has the language model write its answer from what turns up.
The name dates to a NeurIPS 2020 paper by Patrick Lewis and 11 co-authors, first posted to arXiv on 22 May 2020.
RAG supplies knowledge (facts, policies, prices); fine-tuning shapes behaviour (tone, format). A small knowledge base may not need RAG at all.
It cuts hallucinations without ending them. Search can miss, documents go stale and the model can still misread its sources.

What does retrieval-augmented generation mean in simple terms?

A large language model (LLM) such as GPT, Claude or Gemini knows only what was in its training data. It has never seen your refund policy, last month’s price list or the HR handbook on your shared drive. RAG puts a search step in front of the model to close that gap.

The usual analogy is an open-book exam. Without RAG the model answers from memory; with it, someone hands over the relevant pages just before the question. Google Cloud describes RAG as traditional information retrieval (search and databases) combined with generative models, and calls the result grounded generation.

Each word in the name describes one stage:

Retrieval: the system finds the passages in your data that relate to the question.
Augmented: those passages are added to the prompt sent to the model.
Generation: the model writes its reply from the question and the passages it was given.

A 2020 research paper gave it the name

The term comes from Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, written by Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel and Douwe Kiela and listed as a Meta AI (then Facebook AI Research) publication. The team submitted it to arXiv on 22 May 2020, and it appeared in Advances in Neural Information Processing Systems 33 (NeurIPS 2020).

The problem they described will sound familiar to anyone running a company chatbot. Knowledge baked into a model’s weights, which the authors called “parametric memory”, is hard to update and hard to trace to a source. So they paired a text-generating model with a searchable index of Wikipedia (the “non-parametric memory”), reached through a neural retriever. The paper’s abstract reports new state-of-the-art results on three open-domain question-answering tasks, and text that was more specific and factual than a model working from memory alone.

Commercial systems today are a looser version. The 2020 model trained its retriever and generator together, whereas most companies now attach a separate search system to an off-the-shelf model through an API. The principle hasn’t moved: look it up, then answer.

How does RAG work step by step?

There are two phases. Indexing happens once, up front (and again whenever documents change); answering happens every time someone asks a question.

Step 1: Collect and clean your documents

Start with whatever the assistant should answer from, whether help-centre articles, PDFs, policy documents, product sheets or wiki pages. This is the moment to weed out duplicates and old versions. Leave two price lists in the index and sooner or later the system will pull the wrong one.

Step 2: Chunking (split documents into small pieces)

Long documents get cut into short passages called chunks. According to Anthropic’s engineering write-up on retrieval, these usually run to no more than a few hundred tokens (a token is roughly a fragment of a word). Make them too large and each chunk mixes several topics. Make them too small and a chunk can stop making sense by itself, like a paragraph that refers to “this plan” without saying which plan.

Step 3: Embeddings (turn text into numbers)

An embedding model then converts every chunk into a long list of numbers, known as a vector. As OpenAI’s embeddings guide explains, the distance between two vectors indicates how related the texts are, and vectors that sit close together carry similar meaning. OpenAI’s text-embedding-3-small model, for instance, produces 1,536 numbers per chunk by default and accepts up to 8,192 tokens of input. The vectors are stored in a vector database or in a search index that supports vector search.

Step 4: Vector search (find the relevant chunks)

When a question comes in, it is embedded with the same model and the system fetches the chunks whose vectors lie nearest. This is semantic search, and it is why “Can I get my money back?” can match a chunk headed “Refund policy” even though the two share no words.

Meaning-based matching has a blind spot, though. It can miss exact strings like error codes, SKUs or invoice numbers. Many systems therefore run hybrid search, mixing semantic and keyword search, and add a re-ranker to rescore the top results. Anthropic describes pairing embeddings with BM25, an established keyword-ranking method, for exactly this reason.

Step 5: Prompt augmentation (add the chunks to the prompt)

The best few chunks go into the prompt alongside instructions such as: “Answer using only the sources below. If the answer is not in them, say you don’t know. Cite the source title.” The user’s question comes last.

Step 6: Generation and citation

Finally the LLM writes its answer. Since the system knows which chunks it handed over, it can display the source documents beside the reply so that staff or customers can check what they are being told. Making answers traceable was one of the goals of the original paper.

RAG vs fine-tuning vs long context windows: which should you use?

The three get lumped together, but they fix different things. OpenAI’s guide to optimising LLM accuracy draws the line cleanly. RAG is for when the model lacks knowledge, because the information was never in its training data, is out of date or is proprietary. Fine-tuning is for when the model needs to behave differently, for instance by sticking to a format or style.

Long context windows are the newer contender. Current models accept very large prompts: Anthropic’s documentation lists several Claude models that take up to 1 million tokens per request and most others at 200,000 (data as of October 2026). If your material fits, you can skip retrieval and send the entire knowledge base with every question.

Factor	RAG	Fine-tuning	Long context window
What it changes	What the model can see at question time	The model’s learned behaviour	What the model can see at question time (everything at once)
Best for	Large or frequently changing knowledge: policies, catalogues, manuals	Consistent tone, format, classification, specialised output style	Small, fairly stable knowledge bases; analysing one long document
Updating knowledge	Re-index the changed documents	Retrain the model	Edit the text you send
Shows sources	Yes, naturally	No	Possible, if you ask the model to quote
Cost per question	Low (only a few chunks sent)	Low at run time; upfront training cost	High unless prompt caching is used
Main failure mode	Retrieval misses the right passage	Overfitting; knowledge goes stale	Accuracy drops as prompts grow very long
Effort to start	Medium (pipeline, index, evaluation)	High (training data, evaluation)	Low

Long context comes with two caveats. More text does not automatically mean better answers. Anthropic’s documentation calls the slide in accuracy and recall as prompts grow “context rot”, and Lost in the Middle (Liu et al., 2023) found that models do best when the relevant information sits at the start or end of a long input and noticeably worse when it is buried in the middle. The other caveat cuts the opposite way: RAG becomes worth the trouble later than many people assume. Anthropic’s September 2024 retrieval article suggests a knowledge base under about 200,000 tokens can simply go into the prompt, with prompt caching holding down cost and latency.

Nothing stops a company from mixing them. A support assistant might use RAG for facts and a light fine-tune for the brand’s reply format. Mixing can backfire, however. OpenAI’s guide reports a case where adding RAG to a fine-tuned model dropped its score from 87 to 83, because the task was about behaviour and the extra context only added noise.

How much does RAG cost? A worked example

We ran the numbers for a business with a 150,000-token help centre (roughly 190 pages, using OpenAI’s rule of thumb of about 800 tokens a page) fielding 1,000 questions. Prices are list rates for Claude Sonnet 5.5 ($2 per million input tokens, $0.10 per million for cached reads) and OpenAI text-embedding-3-small ($0.02 per million tokens), as of October 2026. Only input tokens are counted, since the written answers cost the same under all three approaches. Rupee figures use the RBI reference rate of ₹94.5975 per dollar for 30 June 2026, the most recent in the RBI handbook table at the time of writing.

Approach	Input tokens per question	Cost per question	Cost for 1,000 questions
Send the full knowledge base, no caching	150,000 × $2/M	$0.30	$300 (about ₹28,380)
Send the full knowledge base, cached reads	150,000 × $0.10/M	$0.015	$15 (about ₹1,419)
RAG: 5 chunks of 800 tokens + 500 tokens of instructions and question	4,500 × $2/M	$0.009	$9 (about ₹851)

A few costs sit outside that table. Embedding the full 150,000-token knowledge base once comes to about $0.003 at the listed price. Writing the cache is charged separately ($2.50 per million tokens for a five-minute cache on Sonnet 5.5, or about $0.375 per write in this example), and because an unused cache expires, a business with little traffic ends up paying for that write more often.

On input alone, RAG works out about 33 times cheaper per question than sending everything uncached. Caching shrinks that advantage to less than two times. At this scale the model bill is modest whichever route you take. What actually costs money with RAG is the engineering and upkeep: a vector database or search service, a pipeline to re-index documents when they change, and hours spent testing answers. For a small business those usually dwarf the token charges.

Where businesses are putting RAG to work

Customer support knowledge base

Support is the most common home for RAG. A chatbot or agent-assist tool searches help articles, resolved tickets and policy pages, then drafts a reply that links to its source. Policies change often, so being able to update by re-indexing rather than retraining counts for a lot here. Many firms start more cautiously with agent-assist, where the AI drafts and a human agent checks and sends.

Internal search across company documents

Staff type a plain question (“What is our travel reimbursement limit for client visits?”) and get an answer pulled from HR policies, SOPs, contracts or the wiki, with the document cited. Permissions are where these projects go wrong. The retriever must search only the documents the person asking is cleared to see; otherwise the assistant turns into a data-leak tool.

Sales enablement

Sales teams use RAG to draw on product sheets, case studies, pricing rules and old proposals when filling in security questionnaires, drafting RFP responses or checking whether a feature comes with a particular plan. Citations let the salesperson verify a claim before a customer sees it. That matters when a wrong answer can harden into a contractual promise.

Other common uses

Onboarding assistants that field new hires’ questions from training material.
Compliance and contract lookup, with a lawyer or compliance officer reviewing what comes out.
Product search on e-commerce sites, where shoppers describe what they want in their own words.

Can RAG still hallucinate?

It can. RAG makes invented answers less likely, not impossible, and the weak spots are well documented.

Retrieval misses. When search fails to surface the right chunk, the model answers from the wrong one or from memory. In Anthropic’s own tests the baseline setup missed the right passage in its top 20 results 5.7% of the time; better embeddings combined with keyword search and re-ranking brought that down to 1.9%. Misses can be reduced but never eliminated.
Irrelevant context. OpenAI’s guide cautions that a pile of irrelevant retrieved text can drown out the useful information and trigger hallucinations.
Stale data. The assistant is only as current as its index. Change a price on the website but leave the old PDF indexed, and it will quote the old price without a flicker of doubt.
Misreading good sources. Given the right passage, a model can still misinterpret it or stitch two passages together wrongly. A peer-reviewed Stanford study of commercial RAG-based legal research tools, published in the Journal of Empirical Legal Studies (2025), found that products from LexisNexis and Thomson Reuters each hallucinated on more than 17% of its test queries.
Messy source documents. Scanned PDFs, tables, slides and images often convert badly to text, so the information never lands in a usable chunk.
Access control and privacy. Indexing sensitive files creates one more place where that data lives, and one more way for it to reach the wrong person.

The remedy is testing. OpenAI’s guide advises building a test set before tuning anything (it suggests starting with 20 or more questions) and measuring retrieval separately from the quality of the final answer. Google recommends tracking metrics such as groundedness. Any vendor promising “hallucination-free” answers deserves a sceptical look.

Should a small business use RAG? A checklist

Answer these before you buy a tool or hire a developer.

Is the problem knowledge or behaviour? If the AI gets facts about your business wrong, RAG is worth a look. If it has the facts but the tone or format is off, work on prompts first and consider fine-tuning after that.
How big is your knowledge base? Estimate the tokens at about 800 a page. Well under 200,000 tokens and rarely changing? Try pasting the lot into the prompt with caching before building anything.
Are your documents in good shape? You want one current version of each policy, clear titles and text-based files. Tidying documents often does more than switching models.
Who owns updates? Put a named person in charge of re-indexing when prices, policies or products change.
Who can see what? Sort documents into public, internal and restricted, and confirm the tool honours those permissions.
Do you have a test set? Write down 20–50 real customer or staff questions along with the correct answers. Run them before launch and after every change.
Will answers show sources? Insist on citations so users can check, and tell the assistant to say “I don’t know” when the sources come up empty.
Is there a human fallback? Refunds, legal, medical and financial questions should go to a person.
Build or buy? Plenty of helpdesk, wiki and cloud platforms now ship document-grounded assistants. Judge them against a custom build using your test set, not the sales demo.
Where is the data stored? Find out where your documents and embeddings are hosted and whether the provider trains on them, particularly if you hold customers’ personal data.

FAQ

What is RAG in AI with an example?

RAG has an AI model look up relevant information before it answers. Say a customer asks a shop’s chatbot how long they have to return an item. The system finds the returns-policy paragraph, adds it to the prompt, and the model answers from that paragraph with a link to it.

Who invented retrieval-augmented generation?

Patrick Lewis and 11 co-authors coined the term in a paper from Facebook AI Research (now Meta AI) presented at NeurIPS 2020. Retrieval-based question answering existed before that, but this paper named RAG and made it popular.

Is RAG better than fine-tuning?

They do different jobs. RAG gives a model up-to-date or private knowledge, while fine-tuning changes how it behaves, such as its output format or style. Some systems use both.

Do long context windows make RAG obsolete?

Not entirely. With a small knowledge base, sending everything in the prompt can stand in for RAG. With large or fast-changing collections, RAG stays cheaper per question and sidesteps the accuracy drop seen in very long prompts.

Do I need a vector database for RAG?

You need somewhere to store and search embeddings, but it needn’t be a dedicated vector database. Plenty of existing databases and search engines support vector search, and many hosted AI tools take care of storage for you.

Does RAG stop AI hallucinations?

No. Grounding answers in your documents makes them rarer, but retrieval can miss the right passage and models can still misread sources. You still need testing, citations and human review.

Salesforce Braze acquisition: $5.66 billion bet on real-time marketing

What is Wi-Fi 7? Real speeds, 6 GHz rules and who should upgrade