What is RAG and why it was Introduced ?

Large language models (LLMs) are probabilistic in nature.

Ask one "How are you?" and it does not search for a stored answer. Instead, it generates the reply one token (a word or fragment of one) at a time.

For example, after writing "I am", it may weigh several possible next tokens:

• "fine" → about 40%
• "good" → about 30%
• "doing" → about 15%

The model samples one of them, appends it to the sentence, and predicts the next token again now conditioned on everything written so far. That is why the same question can produce different responses each time. The answer is generated step by step, not looked up from memory.

Those probabilities come from the model's weights, the billions or trillions of numbers set during training. When training ends the weights stop changing, and everything the model knows is frozen at that date.

So a frozen model cannot learn from being asked questions. If you tell ChatGPT your name today and it greets you by that name tomorrow, it may look like learning, but nothing inside the model has changed. ChatGPT uses the underlying concept of RAG (Retrieval-Augmented Generation) by storing information outside the model, deciding which pieces of information are worth keeping, and retrieving the relevant ones based on your new question before inserting them into the context.

Four things a frozen model cannot do

That storing and retrieving is not just a convenience feature. These mechanisms exist because a frozen model fails in four important ways when used in real world systems.

1. Knowledge cutoff. They cannot access real time information or events past their training date.
2. Hallucination. As we know, LLMs are mathematically optimized to predict the most likely next word, not to verify absolute truth. When they lack an answer, they confidently generate convincing but completely false information.
3. No source attribution. Knowledge is stored abstractly across billions of weights, making specific citations impossible.
4. No access to private data. They are trained primarily on the public internet. They have zero built in access to your organization's private documents, databases, or internal knowledge.

All four limitations stated above share one root cause: the model builds answers from frozen weights. But one part of the model is not frozen. The prompt.

In-Context Learning

LLMs are very good at using new information that is provided in the current prompt.

For example, if you paste your company’s refund policy into a chat and ask for the refund deadline, the model can answer correctly even if that document was never public and the model had never seen it before.

This does not mean the model learned the policy. No training happens, no weights change, and if you start a new chat without pasting the document again, the model no longer has that information.

This behavior is called in-context learning: the model makes predictions from the text available in the current conversation rather than from knowledge stored in its weights during training.

A Survey on In context Learning is the paper that formally defines it.

A natural question is: why not paste the entire company handbook every time?

In practice, that is usually not possible.

The first reason is a hard technical limit called the context window. Every LLM can read only a limited number of tokens in a single request. Many modern models support around 200k tokens, while a 1,000-page enterprise handbook can easily contain 400k–500k tokens. Anything beyond the limit is not read at all.

Even when the document fits, two practical problems remain.

1. Cost and latency grow quickly as the prompt becomes larger, because the entire document must be processed again for every question.
2. Accuracy also drops in very long prompts. The Lost in the Middle study tested whether LLMs could find a specific fact hidden inside a long prompt. Models were much better at finding information near the beginning or end of the prompt, and were more likely to miss important details buried in the middle. As the prompt became longer, overall accuracy generally became worse, even for models designed to handle long contexts.

The best approach is to give the model a short prompt containing only the paragraphs that are relevant to the current question.

This is exactly why RAG (Retrieval-Augmented Generation) was introduced.

• Retrieval: Search a large knowledge base and find only the few paragraphs that are relevant to the user’s query.
• Augmentation: Add those paragraphs to the prompt so the model has the necessary context without overwhelming it.
• Generation: The LLM reads that focused prompt and produces an accurate, grounded answer.

RAG works because it avoids sending an entire knowledge base to the model and instead provides only the small amount of information needed for this specific question.

Follow my newsletter to learn everything about LLMs (internals, training, and more)