What Is RAG: Letting a Model Answer With Your Documents in Hand ​

We · October 2026

This is the eighth article in the "AI Tech Explainers" series, which takes the shape of one word, properly explained, per article: no product reviews, no rankings, no download or sign-up links. Article two of the series covered the four layers—engine, runtime, client, interface apps—and answered how a model gets running. The layer described here sits on top of those four and answers a different question: what makes the model know your stuff. It pairs with What Is MCP? Why It Became the Connection Standard for AI Tools: one hands material to the model, the other hands it tools.

AI moves fast. The tools, capabilities and numbers in this article describe where things stand right now, and are for reference only. Model information was checked in October 2026; for actual features and license terms, each project's own site and its code and model repositories are the authority.

First, separate the three problems people mix together ​

When a model cannot answer something written in your company's manual, the usual explanation is that it has a bad memory. That explanation sends your tool choice in the wrong direction, because what is actually stacked there are three problems with nothing in common.

Freshness comes first. Once training ends, the weights freeze where they are; the model does not know what happened afterward. Your material, meanwhile, changes every day—a new policy, a new version number, the meeting notes from last week. Re-teaching that batch into the model is a completely different kind of cost, and one that is often outside what an individual can do at all.

Traceability comes second. In some settings a correct answer is not enough: you have to be able to point at the page a conclusion came from. Contracts, medical records, the definitions behind a financial report, the rules a customer quote is priced by—all of those. And a document that says the wrong thing versus a model that answered crookedly are two different responsibilities, which have to stay separable. Pushing the whole pile of material into context is precisely what makes them impossible to tell apart.

Cost and latency come third, and this one is the most concrete. Sending an entire manual to the model on every question, versus pulling only the few most relevant passages, are not in the same order of magnitude—in money or in waiting time. Even when the window can hold the whole thing, the bill still does not make sense.

RAG (Retrieval-Augmented Generation) targets those three: it first finds the relevant passages in your material, then hands them to the model along with the question and lets the model answer from what was retrieved. One sentence fixes where it sits—RAG does not make the model smarter; it gives the model a reference book it is allowed to flip through. If the material does not cover a question, it still gets that question wrong; if the material is wrong on the page, it repeats the error. Both of those happen outside the chain, not inside it.

One chain, four steps, each with its own trap ​

Of these four steps, the first is what most people underestimate. If the text extracted from a PDF comes out in the wrong order, the other three steps—however careful—are just precision arithmetic performed on garbage. This is also why document parsing is its own category in the local tool list: tools like MinerU exist to turn PDFs, images and ebooks into clean Markdown or JSON. The size we recorded in the list is just over 17GB, with 16GB of memory recommended, and on Windows a GPU is the suggestion—heavier than many people expect, because layout recognition has models of its own inside it. And the output of this step is reusable: the Markdown file you produced outlives every vector database you will ever build. Swap the embedding model, swap the software, swap the wrapper front end—that file does not have to be redone.

Progress on the chunking side has been concrete over the past two years. The flaw in the traditional cut is that a passage detached from its context tells nobody what it is about. Read alone, "the plan is expected to bring a 15% improvement" names neither which plan nor an improvement in what. The accepted practice now is to attach, at index time, one line to every chunk saying which document and which chapter it came from (the family of approaches the industry calls contextual retrieval). The cost is slower ingestion and more storage; the payoff is noticeably steadier recall. The plain substance of it: an index should hold not just the words but the identity papers of each passage.

The embedding layer is the part you can genuinely handle locally ​

Retrieval runs on the small model that turns text into numbers, and in 2026 the open-source choices at this tier are plentiful—and far lighter than most people assume.

Two families come up repeatedly in the public material. One is the BGE series from BAAI: BGE-M3 supports over a hundred languages, takes more than eight thousand tokens for a single passage, and emits dense, sparse and multi-vector representations at once, which lets it do hybrid semantic-plus-keyword retrieval in one pass. The other is the Qwen3-Embedding / Qwen3-Reranker tier released by Alibaba, in 0.6B, 4B and 8B sizes, with public claims of support for dozens of languages; the smallest tier's weights are only a few hundred MB and run on CPU. Leaderboard scores are not comparable across sources, so they are not worth much; what an individual user actually applies is this: start with the smallest tier, it is usually enough; move up in size only when it is not.

Two traps at this layer are worth knowing in advance. One: an embedding model is not something you swap casually. The numbers in a vector index mean something only under the model that produced them, so a new model means re-embedding the entire library. Somebody has publicly recorded a full re-embed of a Chinese library of tens of thousands of memories taking three or four hours (a community record, checked October 2026, useful as an order of magnitude). Choosing this model is choosing the one link in your setup you cannot move lightly for the next six months. Two: mixed Chinese and English text, along with numbers, model designations and code identifiers, are the shared weak spot of this layer. For a question like "what exactly is this model called", pure semantic similarity regularly loses to the dumbest keyword match—which is why hybrid retrieval with a keyword path has come back as standard equipment in the past two years.

Reranking is the other half of the answer. The retrieval stage has to be fast, so it lets in a large batch of passages that are "possibly relevant". The reranker then works through those candidates one at a time against the question and squeezes the list down to the top few. It is not an optional extra—RAG that skips the rerank is the model guessing, inside a pile of seventy-percent-relevant material, which one is actually on point. And faced with contradictory material, a model usually serves up the reading that flows best, not the one that is right. The open-source tier has the same kind of choice here—BGE-Reranker, Qwen3-Reranker and their peers, mostly under Apache 2.0, though the LICENSE in each repository stays the authority.

GraphRAG, which gets mentioned alongside all of this, is positioned to fill a different gap: vector retrieval is good at "which passage looks most like this question" and poor at "assemble one causal chain across several documents". Turning material into entities and relations and retrieving over that graph does handle these multi-hop questions. The price is a build stage that runs a model over the entire corpus once more—expensive—and a graph that stops keeping up with your material as soon as it is finished. For a personal knowledge base the odds of hitting that kind of problem are low; do not go and build one before you need it.

With contexts this long, do you still need RAG? ​

This is the question asked most often in 2026, and the answer is more complicated than either yes or no: long context redrew the line between the two jobs, it did not retire retrieval. There are three reasons, and each has public measurements to check against.

Fitting it is not the same as using it. Windows on mainstream models already sit in the 128K-to-a-million-token range, but when the key information is placed in the middle of a very long context, recall degrades noticeably. That is an old habit of attention mechanisms and no model escapes it; a high score on a "needle in a haystack" test does not mean every passage of a long document gets read reliably. A window is a capacity number, not a reliability number.

Cost and latency are hard constraints. Processing a hundred thousand tokens and processing ten thousand are not in the same order of magnitude, in price or in waiting time, and attention computation grows worse than linearly. The public enterprise-level comparisons point the same direction: same set of documents, same daily query volume, a retrieval-based setup comes out one to two orders of magnitude cheaper than feeding whole documents in, with lower latency (mostly figures published by the projects themselves or by third-party self-tests, checked October 2026, useful as an order of magnitude). For readers running models locally, this one is more direct still: the longer the context, the bigger the key-value cache your card has to hold at the same time—which is exactly the arithmetic article three of the series worked through.

Knowledge updates and permissions are not the window's job. An index updates incrementally: add one new meeting note today and the next retrieval returns the latest version. Re-stuffing documents into the prompt means redoing the whole thing every time. And a boundary like "this material is readable by that department only" is certainly not something context solves.

So the more common setup in 2026, and the cheaper one, is two-stage: use cheap retrieval to squeeze a large pool of candidates down to a short stretch, then let a long-context model compare and synthesize inside that stretch. With that framing, the routing rule is easy to apply: one document, asked and answered in a single sitting—drop it in the client's attachment box and do not build an index. Dozens of documents, updated daily, needing a citable source, or simply not wanting to paste the whole thing again every time—that is where RAG belongs.

Three forms, and you are already using the first one ​

The product shapes at this layer have already split cleanly. Our app list carries examples of all three, and the gap between what they ask of your machine is itself instructive.

Put the three side by side and something counter-intuitive shows up: this layer does not eat GPU, it eats disk, document quality, and your own tidying. In that same list, the clients that run large models advise reserving over 100GB of disk, while the knowledge-base orchestration tools start up with a few GB—because a vector index is text plus a string of numbers. It does not need compute piled on top. It needs the material turned into text properly, cut along its own paragraphs, and somebody in charge of keeping it updated.

One more thing to say plainly, since licenses come up here constantly: GPL and AGPL appear very often among the terms these platforms use, and a few others use licenses where the source is public but that is not the same as free to use (n8n, with two licenses listed side by side, is the typical case). Running it on your own machine and packaging it as a service you sell to someone else are two different acts, so do not let the phrase open source stand in for a commercial-use check; the LICENSE in the repository decides everything. This is the same principle as Where Model Files Come From: Repos, Community Quants, and Licenses.

RAG and MCP are two different things ​

The article before this one, on MCP, gets read as "yet another way to connect my documents". The two sit on opposite sides of one chain.

RAG handles material you already own: you organize it, index it, and when you ask a question it puts the relevant passages in front of the model. The boundary of the answer is set by your library. MCP handles the interface between the model and the outside world: it lets the model call a set of tools and data entrances somebody else has already defined, and what comes back depends on what the server on the other end is connected to. The first is a library you built; the second is a door someone else opened. If the job is "find the answer inside these documents of mine", that is a retrieval problem. If it is "ask that system, and get something done while you are at it", that is what an interface is for.

Three questions to size it up ​

1. Is my material a one-off, or something I will keep querying? One file, a few rounds of questions and it is done: the client's attachment feature is enough, do not build an index. Dozens of files, changing daily, with sources to cite—that is when the retrieval layer comes into play.
2. Can my material be turned into clean text? Scans, two-column layouts, table-heavy reports: solve parsing first, or every later step just amplifies the mistake made at this one.
3. Can I accept that it will still get answers wrong? Retrieval guarantees only that the model saw your material. It does not guarantee the material is correct, or that it covers the thing you asked about. So whether you can see the cited source, and get back to the original document in one click, should be a hard requirement in your selection—it decides whether you can find the cause when the answer is wrong.

What RAG produces is not an answer, it is material you can re-check ​

This layer reaches the same conclusion as the article on speech AI: what is genuinely yours after a run is not this session's answer, it is the material you tidied. Answers go stale. A clean Markdown file, a chunking scheme you keep using, an index whose embedding model you know the name of—those are assets you can carry with you.

Our view is that the payoff here for individual users is narrower than engineers imagine. You most likely do not need a graph, do not need multi-path recall, and do not need candidates piled up by the dozens. Turn the material into orderly text, cut it along its natural paragraphs, keep the sources attached, and run the whole thing once through the smallest embedding model. Do those four, and local RAG already covers most of the "how can it possibly not know my thing" cases. Whether it deserves a platform of its own comes down to how often your material changes, and whether you are willing to pay the hours of a full re-embed when you do swap models.