In short
Retrieval augmented generation (RAG) is an architectural pattern for AI applications in which a large language model (LLM) is specifically supplied with matching information from an external knowledge source before it generates an answer. Instead of relying solely on the knowledge frozen into the model during training, the system looks up the relevant documents for each request (retrieval), passes them to the model as context (augmentation) and only then lets it formulate the answer (generation).
The result: answers based on current, verifiable and company-owned data rather than on whatever the model once learned at some point.
The term goes back to a research paper by Patrick Lewis and colleagues from 2020 ("Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", Meta AI). What was an academic idea at the time is today the standard pattern with which companies integrate their own data into AI applications securely and up to date — from the internal knowledge assistant to the customer service chatbot. When people talk about "RAG AI" in a business context, it is exactly this combination of search and language model that is meant.
What is retrieval augmented generation and what problem does RAG solve?
Language models have three structural weaknesses that quickly become expensive in business use. First, their knowledge ends at the training cut-off date — the model knows nothing about what happened afterwards. Second, they know no internal data. A company's price lists, contracts, product documentation or support history were never part of the training. Third, when knowledge is missing, they invent plausible-sounding answers — the well-known hallucination problem.
RAG addresses all three points with the same mechanism: the model does not have to know the answer, it only has to put it together from the sources supplied. The knowledge base can be updated at any time without retraining the model. Internal documents remain in your own infrastructure and are only passed to the model as context. And because the answer is based on specific passages of text, the system can cite its sources — which does not eliminate hallucinations completely but reduces them significantly and, above all, makes them verifiable.
How RAG works: architecture in four steps
Reduced to one sentence, every RAG application runs in four steps:
- Ingestion and chunking: source documents are read in, cleaned and split into meaningful sections.
- Embedding and vector index: each section is translated into a meaning vector and stored in an index.
- Retrieval: for each user question, the sections that best match in content are fetched from the index.
- Generation with context: the language model formulates the answer exclusively on the basis of the sections found – ideally with a source reference.
The first two steps form the preparation of the knowledge base (indexing), the last two the answering of a request (retrieval and generation).
Phase 1: building the knowledge base
First, the source documents — PDFs, wiki pages, product data, support tickets — are split into manageable sections, known as chunks. An embedding model translates each chunk into a numerical vector that represents its meaning. These vectors end up in a vector database that handles semantic similarity search in milliseconds. Texts that are related in content lie close together in this vector space — even if they do not share a single word.
Phase 2: request, retrieval, answer
When a user asks a question, it goes the same way: the question is translated into a vector, the vector database returns the chunks most similar in content, and these passages are embedded together with the original question in the language model's prompt — this is where the disciplines of RAG and prompt engineering come together. The model formulates the answer exclusively on the basis of the context supplied, ideally with a source reference. Modern systems add further stages to this core: hybrid search (vector plus keyword search), re-ranking of the hits and filtering by access rights.
RAG vs fine-tuning – or a large context window?
RAG is not the only way to give a model additional knowledge. But the three common approaches solve different problems:
| Approach | Principle | Suitable for | Limits |
|---|---|---|---|
| RAG | Knowledge is retrieved and supplied at runtime | current, extensive, changing knowledge; source citations; access rights | Quality stands and falls with retrieval |
| Fine-tuning | Model weights are retrained with your own data | tone, format, technical language, recurring task patterns | expensive, static, no source reference, update = new training |
| Large context window | all documents are packed into the prompt in full | small, self-contained sets of documents | costs and latency rise with every token; impractical for large collections |
In practice the approaches are not mutually exclusive: a fine-tuned model can run in a RAG architecture, and for small amounts of knowledge the large context window is often the simplest solution. As a rule of thumb: if it is about factual knowledge that changes, RAG is the right lever. If it is about the model's behaviour, fine-tuning is the right one.
RAG vs fine-tuning compared directly
| Criterion | RAG | Fine-tuning |
|---|---|---|
| Topicality | Knowledge base can be updated at any time, no retraining | State of the last training run; update = new training |
| Data control | Documents remain in your own database, can be deleted and filtered | Data moves into the model weights, cannot be deleted selectively |
| Costs | Effort mainly in data preparation and retrieval quality | Training runs and GPU time, again with every change |
| Traceability | Source reference possible for each answer | No reference to individual sources |
| Use | Factual knowledge that changes; internal documents; access rights | Tone, format, technical language, recurring task patterns |
A real-world example: a support assistant in e-commerce
A mid-sized online retailer with around 40,000 items wants to relieve its customer service. A classic chatbot regularly fails there: questions such as "Does spare part X fit model Y from 2019?" cannot be covered with pre-written answers, and a pure language model would guess for lack of product knowledge.
The RAG solution: product data sheets, compatibility lists, shipping and returns policies and the most frequently solved support tickets are split into chunks, embedded and indexed. If a customer asks about compatibility, retrieval finds the matching compatibility list and the associated data sheet, and the model formulates a concrete, verifiable answer from them — including a link to the source. If a return period changes, only the document concerned is re-indexed; five minutes later the assistant answers correctly. No retraining, no code change. It is exactly this property that makes RAG attractive for mid-sized companies: knowledge maintenance stays where it already takes place today — in the documents.
Typical pitfalls in RAG projects
As elegant as the pattern is, many projects fall short of expectations. The most common causes almost never lie in the language model but in data preparation and retrieval:
- A poor chunking strategy: if documents are bluntly cut every 500 characters, tables, lists and meaningful contexts are torn apart. Retrieval then finds fragments the model can do nothing with.
- Outdated or contradictory sources: RAG makes the knowledge base the single source of truth. If it contains three versions of the same price list, the system will eventually quote the wrong one. Data hygiene is not a preliminary project but part of operations.
- Missing permission checks: retrieval must respect access rights. An assistant that hands out salary lists to anyone who asks cleverly is a data protection incident waiting to happen.
- No handling of "not found": if retrieval delivers nothing useful, the system must say so — instead of letting the model fantasise freely. An honest "I have nothing on that" is half the value of a RAG application.
- Missing evaluation: without a test set of real questions with expected answers, it is impossible to measure whether a change to chunking, the embedding model or the prompt improves or worsens quality.
Data protection and GDPR: why RAG scores in mid-sized companies
An often underestimated advantage of RAG lies in data protection. With fine-tuning, company data moves permanently into a model's weights — once trained in, it can neither be deleted selectively nor made available for access requests. With RAG, by contrast, the documents remain in a normal database under your own control. Erasure obligations under Art. 17 GDPR can be met by removing the document concerned from the index; access requests can be answered because it is traceable which sources are stored. Content only goes to the language model as context on a per-request basis — with an EU-hosted provider with a data processing agreement or a self-hosted model, this makes a clean, documentable data flow possible. For companies with sensitive data, it is precisely this separation of knowledge and model that is often the decisive argument for RAG.
Variants: from naive RAG to GraphRAG
Several stages of development have become established under the umbrella term. Naive RAG refers to the basic form: embed, search, answer — quickly built but prone to weak hits. Advanced RAG adds optimisations before and after retrieval: requests are reformulated or split into sub-questions (query rewriting), hits are re-sorted by a re-ranking model and metadata filters (date, department, language) are applied. GraphRAG combines vector search with a knowledge graph that maps relationships between entities — useful when answers require connections across many documents, such as "which customers are affected by the change to product X?". For most mid-sized projects, advanced RAG is the realistic target state; GraphRAG only pays off with complex, highly interlinked data.
RAG and AI agents
In modern architectures, RAG is rarely a stand-alone system but a building block: AI agents use retrieval as one of several tools. Instead of every request rigidly going through the same retrieval path, the agent decides itself whether and which knowledge source to consult — this is then called agentic RAG. The agent can repeat a search several times with reformulated queries, combine results from different sources and research further when there are contradictions. The retrieval logic thus moves from a rigid pipeline design into the agent's planning loop — with all the opportunities and control requirements that agent systems bring with them.
How much set-up effort is realistic depends almost entirely on the state of the source data. A well-maintained Confluence wiki or structured product data can be indexed in a few days. A historically grown filing folder with scanned PDFs, duplicates and outdated versions, on the other hand, first needs a clean-up round — and that is exactly where it is decided whether the later application delivers trustworthy or random answers. An honest assessment of the data situation therefore belongs at the beginning of every RAG project, not at its end.
Conclusion
Retrieval augmented generation is the most pragmatic way to connect language models with current and company-owned knowledge: updatable without retraining, citable, and privacy-friendly because the documents remain in your own infrastructure. Success depends less on the choice of model than on matters of craft — data quality, chunking, the permissions concept and evaluation. Anyone who builds these foundations properly gets, with RAG, the building block that turns a generic language model into an assistant with real company knowledge — a core task of any well-founded AI consulting.
Frequently asked questions about retrieval augmented generation
Does RAG prevent hallucinations completely?
No, but it reduces them significantly and makes them verifiable. The model can summarise the supplied sources incorrectly or mix in its own "knowledge" despite the context. Source citations, a strict "answer only from the context" in the prompt and defined behaviour when nothing is found are therefore mandatory.
Does every RAG system need a vector database?
For semantic search, a vector database is the standard, but not a dogma. Small collections also work with classic full-text search or a combination of both (hybrid search). What matters is that retrieval finds the passages that are actually relevant — which technology it uses is secondary.
What does it cost to run a RAG application?
The running costs are made up of embedding and LLM API calls (or your own model infrastructure), operating the vector database and data maintenance. For typical mid-sized scenarios, the pure model costs per request are in the cent range; experience shows that the bigger item is the initial data preparation. An experienced AI agency helps to estimate precisely this effort realistically and to choose the right RAG architecture.
Is RAG GDPR-compliant?
RAG is not a product but an architectural pattern – whether a specific application is GDPR-compliant is decided by hosting and data flow. Because the documents are stored in your own database and only go to the model as context on a per-request basis, the data flow can be documented well and restricted to EU hosting or a self-operated model. You meet erasure obligations via the index, not via retraining. A data processing agreement with the model provider remains mandatory as soon as personal data ends up in the context.
Do I need my own model for RAG?
No. RAG works with any current language model – via the API of an EU-hosted provider or with an open model on your own infrastructure. The company knowledge sits in the knowledge base, not in the model; the model only has to be good at answering from the context supplied.
RAG or fine-tuning – which first?
In most business projects, RAG first. It is quicker to set up, keeps the knowledge current and provides source references. Fine-tuning only pays off when the model is supposed to answer differently on a permanent basis – in a particular tone, format or jargon – and even then usually remains an addition within a RAG architecture.
Want to make company knowledge usable with RAG? Then have RAG knowledge assistants developed – with permission checks, source citations and EU hosting.