RAG: how AI answers on top of your own data without retraining a model.
The mechanism in four steps, what really decides the quality of the answer, and the two situations where RAG is the wrong tool and somebody finds out late.
The short answer.
RAG is the model consulting a base that belongs to you before it answers. AWS describes it as the process of optimizing the output of a language model so that it references a trusted knowledge base outside its training sources. The model stays the same; what changes is what it reads before it writes.
The acronym comes from a 2020 paper by Patrick Lewis and colleagues, presented at NeurIPS. There, models combine two memories: the parametric one, held in the model's weights, and the non-parametric one, an index consulted at the moment of answering. Every RAG implementation in use today is a variation on that idea.
For a company, the practical consequence is direct: the model can answer about internal policy, contracts, a catalogue and history without sending any of it into a training run, and without waiting for the next version of the model.
The four steps.
This is the sequence AWS describes, in the order it happens on every question. The fourth step is the one most projects forget.
Prepare the external base
The documents that sit outside the model's training are converted into numerical representations and stored in a vector database. This is the base the model consults, and it is yours, not the model vendor's.
Retrieve what is relevant
The question also becomes a numerical representation and is compared against the base. The closest passages come back by similarity, not by keyword.
Augment the prompt
The retrieved passages go to the model alongside the original question. The model answers looking at that context, instead of answering only from what it memorized in training.
Update the base
Documents and their numerical representations are refreshed automatically or in batches. Without this step the system ages in silence: it keeps answering confidently, now about the old version of the policy, the price or the contract.
The question that decides everything: what unit are you indexing?
Before choosing a model, choose the chunk. If the indexed unit is a blind slice of a thousand characters, retrieval returns half a sentence from one clause and the start of another — and the answer comes out wrong with every appearance of being right.
A chunk with meaning
A clause, a section, a catalogue item, a support reply. The unit has to be something a person would quote whole, not a fixed length.
The source stuck to the chunk
Document, version, page and date travel with the passage. Without them the answer cannot show where it came from, and nobody audits anything afterwards.
Permissions at retrieval
Who can see what is filtered before the passage reaches the model. Asking the model not to tell is access control based on good will.
A set of real questions
Thirty questions the business actually asks, with the right answer written beside each. It is what turns "it got better" into a number, and it is what almost nobody builds first.
When RAG helps, and when it does not.
The pattern is good for one class of problem and terrible for another. Knowing the difference in advance saves a quarter.
- A question whose answer is written in some company document, and nobody can find the document.
- Knowledge that changes every week and cannot wait for a new version of a model.
- Support that has to cite the current policy and show the passage it used.
- Triage: reading a lot of material and pointing at what a person should look at first.
- A number. Balances, prices, scores and deadlines come from a deterministic, tested calculation, with the model writing only the text around it.
- An answer that depends on live state — stock right now, order status right now. That is a system call, not document retrieval.
- A base nobody maintains. If the source document is out of date, RAG delivers the error faster and with more conviction.
- A decision that must always be the same for the same input. A rule is a rule: write it as a rule.
Questions about RAG.
What is RAG?
AWS defines RAG as the process of optimizing the output of a language model so that it references a trusted knowledge base outside its training sources before generating a response. The term comes from the paper by Lewis and colleagues, presented at NeurIPS in 2020, which described models combining parametric memory, learned in training, with non-parametric memory retrieved from an index.
Is RAG the same as training the model on my data?
No. Training changes the weights of the model; RAG does not touch the model and changes the base it consults. AWS points to cost as the main reason to prefer RAG: retraining a foundation model on organization-specific information is computationally and financially expensive, and RAG introduces new information without that.
Does RAG eliminate hallucination?
It reduces it, it does not eliminate it. The model now answers over a context you control, which cuts out the class of error where it invents an answer because it has no information. It can still misread the retrieved passage — which is why the answer has to show its source, so a person can check it in one click.
What decides the quality of a RAG system?
Retrieval, almost always. If the right passage does not come back from the base, no model saves the answer. What moves that needle most is the size and shape of the indexed chunk, what goes into the base, and what is refreshed how often — none of which is a choice of model.
When is RAG the wrong tool?
When the answer is arithmetic rather than text. Balances, prices, scores and deadlines should come from a deterministic, tested calculation; the model writes the text around the number, never the number. It is also the wrong tool when different people may see different parts of the base: permissions then have to be applied at retrieval, not asked of the model.
Do I need a vector database to do RAG?
You need some way of retrieving the right passage. A vector database is the most common implementation because it compares meaning rather than exact words. On a small, well-structured base, a well-built conventional search does the job and is cheaper to run.
Want to know whether your own base can take this today?
In most companies the answer is no — and the reason is not the model, it is the data. The diagnostic looks at what you have, says what has to change first, and what each part costs.
Sources
The definition and the four steps come from the AWS page; the origin of the term comes from the original paper. Both were opened on August 9, 2026. Everything else is implementation practice and is described as practice, not as a citable fact.