Two million tokens did not remove the need for retrieval
Choose long context or retrieval by the documents, permissions and update rate your assistant must handle. A larger window alone does not settle the decision.
By 
Paul. AI and agentic engineering
An assistant that can accept the company handbook still needs a way to decide which handbook, which revision and which reader it is serving. That is why large context windows did not settle the argument about retrieval. For business assistants, document selection remains a product responsibility even when an entire collection fits into one prompt.
The useful decision is about the work. Long context suits questions that need the relationships across a bounded set of documents. Retrieval suits repeated questions over a changing collection, especially where different people have different access. Neither choice guarantees a correct answer. Looking back from November 2026, the 2024 announcement helps separate capacity from responsibility.
What changed in June 2024
On 27 June 2024, Google opened Gemini 1.5 Pro’s two-million-token context window to all developers. The same announcement described context caching for Gemini 1.5 Pro and 1.5 Flash as a way to reduce the cost of repeatedly using the same input tokens. The earlier two-million-token announcement had been behind a waitlist. This was wider access, not the first mention of the capability.
Those facts support a narrower conclusion than “retrieval is obsolete.” More material could be supplied in one request, and repeated input had a new cost mechanism. They did not show that every question needed the whole collection or that a model would reliably locate every relevant passage. The announcement was a capability and product update, not an evaluation of your documents.
The Gemini team’s technical report, version 3 dated 14 June 2024, provides research context for the long-context capability. Its evaluations remain the model maker’s evaluations, not acceptance evidence for your assistant.
This is a retrospective design argument, not a recommendation to buy a 2024 model today. No current tariff, supported model list or caching lifetime is assumed here. Those details must be checked for the service you actually choose.
Whole-document work is a real use case
Consider an illustrative task: compare the responsibilities in a small set of contract drafts that a reviewer is authorized to read. A clause in one section may qualify a promise elsewhere. A retrieval system that selects isolated passages can omit the qualification. Giving the model the complete bounded set may be a simpler starting point.
That example does not establish legal accuracy or replace professional review. It shows why “always retrieve the smallest chunk” is also too rigid. If the question depends on the shape of the whole document, fragment selection becomes an additional source of error. You would need to evaluate whether the selected context retains what the question requires.
A useful prototype can therefore start with whole documents and explicit document identifiers. Keep the input set fixed, record its revisions and require the answer to identify supporting passages. Then check whether the passages support the conclusion. An answer that looks like a citation is not evidence until somebody or an appropriate check validates the relationship.
A document collection has rules beyond size
Now consider a different illustrative assistant: employees ask about internal procedures while managers also have access to restricted commercial documents. Sending the entire collection to every request would put unauthorized material into the model’s input. An instruction to avoid mentioning it is a poor substitute for withholding it.
Retrieval can select material after the application has checked access. That is an architectural opportunity, not an automatic property of a vector database. Permissions must constrain selection before generation; filters need verification, and any shared caches must respect the same boundaries. A whole-document design can also be secure when the application selects only documents the requester may read.
Changing information creates a similar obligation. A passage selected from an old index can be stale. So can a complete document retained in a cache. The design needs revision identifiers and a rule for replacing or invalidating old material. Neither “RAG” nor “long context” tells you who owns that rule.
The original Google announcement explicitly linked caching to reuse of the same tokens. The design inference is straightforward: repeated stable input is a different workload from constantly revised input. Evaluate them separately rather than applying one optimistic cost estimate to both.
Compare the cost of an accepted answer
Retrieval introduces work: preparing documents, finding relevant material and maintaining that preparation as the collection changes. Whole-context requests can avoid some of that machinery. They may also process much more input than a particular question needs. A cache discount alone cannot determine which system costs less overall.
For a fair comparison, use representative questions and include document preparation, model requests, cache-related charges where applicable and human correction. Compare accepted answers, not the cheapest single request. Record latency separately; a cheaper answer that arrives too late for the task may still be unsuitable. These are proposed evaluation criteria, not results from an experiment conducted for this article.
Include failures that each approach could hide. Ask a question whose answer crosses sections. Ask one that has no answer in the supplied material. Replace a document and repeat the question. Run an access test with two users who have different permissions. These cases reveal more about the product than a demonstration built around a convenient question.
Start with the boundary, then choose the mechanism
The strongest objection to retrieval is operational complexity. For a modest, stable, uniformly accessible collection, maintaining an index may cost more effort than it saves. In that situation, whole-context requests deserve a serious trial. The strongest objection to whole context is that a growing window can conceal poor document selection until cost, freshness or permissions fail.
A hybrid is reasonable when the work calls for it: select the authorized document set, then provide enough complete context for the question. It earns its complexity only if the evaluation shows a useful improvement.
Before commissioning an AI assistant, specify the collection, its change rate, the permission boundary and the evidence an answer must carry. Window size becomes a constraint within that decision. It cannot make the decision for you.