Whole-Document Context or Passage Retrieval?

The choice between putting a whole document in a model’s context window and retrieving selected passages is a decision about evidence coverage, reliability, and repeated cost. Whole-document context makes the entire text available to the model in one request. Passage retrieval selects smaller portions from an indexed corpus and supplies them with the question, typically in a retrieval-augmented generation (RAG) workflow.

Use whole-document context when the answer depends on relationships across sections, implicit evidence, open-ended synthesis, or a sequence of reasoning that cannot be localized confidently. The document must also fit within the model’s practical context and cost limits, not merely its advertised maximum window.

Use passage retrieval when questions are targeted, the corpus is large, or the same material will be queried repeatedly. Retrieval can reduce the text sent for each answer, but it introduces a separate failure point: the system may omit or misrank the evidence needed to answer.

A hybrid route is often a useful starting point: retrieve first, assess whether the passages support the answer, and escalate to the whole document when evidence is incomplete or the question requires cross-section reasoning. This is a testable design choice, not a guarantee that the hybrid system will be best for every task.

The practical context limit is smaller than a document-size limit

A document’s token count is only one part of the input budget. Instructions, the user’s question, document formatting, citations, and reserved output space also consume capacity. A document that technically fits may still be impractical if it causes excessive latency, expense, or degraded performance.

Even when the entire document is present, the model may not use all parts equally well. Controlled multi-document question-answering and key-value retrieval experiments found that performance often declined when relevant information appeared in the middle of a long prompt, relative to positions near the beginning or end [1]. That finding does not establish the same behavior for every current model or task, but it does show why a large context window should not be treated as proof of reliable attention to every passage.

How the Two Workflows Handle Evidence

Whole-document prompting

A whole-document workflow supplies the document and a task instruction together. It is especially appropriate when the scope of relevant evidence is difficult to predict in advance.

For example, a request to explain how a policy’s exceptions interact with its definitions and enforcement provisions may require evidence from several distant sections. An open-ended summary may likewise need to represent the document’s structure rather than answer a single localized question.

A reliable implementation should:

1. Preserve section headings, tables, and other structure needed to interpret the text.
2. Give the model a specific task and ask it to identify supporting sections or passages.
3. Reserve sufficient space for the answer and any citations.
4. Test questions whose evidence appears at the beginning, middle, and end of representative documents.
5. Check whether the answer overlooks qualifications or combines statements that apply under different conditions.

Whole-document prompting reduces dependence on an initial retrieval step, but it does not remove the need to verify citations or factual support. It can also become expensive when the same document is sent repeatedly for many questions.

Passage retrieval

A retrieval workflow first prepares the source material for search. The document is divided into chunks, which are indexed using semantic representations, keyword search, or both. At query time, the system retrieves candidate chunks, may rerank them, and sends selected passages to the model with the question.

The design choices matter. Chunks that are too short can lose definitions, exceptions, or the identity of the entity being discussed. Chunks that are too long may dilute the relevant evidence and consume more of the context budget. Metadata such as document title, section heading, version, and access permissions helps the system interpret and govern retrieved material.

Retrieval should be improved before concluding that a task necessarily requires full context. In Anthropic’s Contextual Retrieval approach, a short document-aware explanation is prepended to each chunk before indexing; the workflow combines embeddings with BM25 keyword search and can use a reranker [3]. In Anthropic’s evaluation, top-20 retrieval failure fell from 5.7% for the baseline to 3.7% with contextual embeddings, 2.9% with contextual embeddings plus contextual BM25, and 1.9% after reranking [3]. Those figures concern retrieval failure in that evaluation, not final-answer accuracy or expected gains on another corpus.

A passage-based answer should therefore be judged on two separate questions: Did retrieval surface the necessary evidence? And did the model use that evidence correctly? A fluent answer does not establish either.

Comparing Whole Context, Retrieval, and Hybrid Routing

Where each method is most defensible

Neither method wins universally. A study comparing RAG and long-context prompting across public datasets found that long-context prompting generally performed better when adequately resourced, while RAG used fewer input tokens [2]. Its Self-Route method checked whether retrieved passages could answer a query, accepting supported RAG cases and routing others to full context. In the study’s evaluations, Self-Route approached long-context performance while using 61.4% of long-context tokens with GPT-4o and 38.4% with Gemini-1.5-Pro [2]. Those are results for the evaluated systems and datasets, not cost or quality guarantees for current deployments.

Build an evidence-based escalation rule

A hybrid system needs a more precise rule than “retrieve, then trust the model.” A practical workflow is:

1. Retrieve candidates. Search for the question using semantic and keyword signals where appropriate.
2. Check evidence coverage. Determine whether the passages support every material part of the proposed answer, including relevant conditions and exceptions.
3. Answer from supported passages when coverage is sufficient, citing the source locations.
4. Escalate when passages conflict, omit necessary context, or the question requires relationships across sections.
5. Abstain or request more material if the full document is unavailable, too large for the practical budget, or still does not establish the answer.

The coverage check itself can fail. A model may be confident in an answer based on incomplete passages, and retrieved passages cannot prove that an omitted exception does not exist elsewhere. Evaluation should therefore include cases where a superficially relevant passage is incomplete or contradicted by another section.

AI Plaza, an active industry participant with a model catalog, provides a practical setting for comparing these routes without assuming that one model or method is preferable. For example, an evaluation could run GPT-6.1 Sol, Claude-Sonnet-5.5, and Gemini-4-Argon on the same representative questions: retrieve first, check passage sufficiency, and escalate eligible cases to whole-document context. The model names should be taken from the product catalog and placement overrides at publication time rather than hardcoded as a permanent default stack.

Evaluation, Costs, and Long-Term Implications

Measure answer quality and system behavior separately

A useful evaluation set includes targeted lookups, cross-section questions, open-ended synthesis, ambiguous questions, and questions the document cannot answer. It should also include evidence at different document positions and examples with exceptions or conflicting passages.

Track at least four outcomes:

• Answer accuracy: whether the substantive answer is correct and appropriately qualified.
• Evidence coverage: whether cited passages support each material claim and whether necessary qualifications are included.
• Token use and cost: input and output consumption across retrieval, checking, escalation, and generation—not just the final answer request.
• Latency: time spent on search, reranking, model calls, and any fallback.

Also record retrieval misses, unsupported answers, unnecessary escalations, and cases where the system should have abstained. Report results by question type: an aggregate score can conceal a system that handles simple lookups well but fails on cross-section dependencies.

Repeated-query economics deserve particular attention. Whole-document prompting may be straightforward for occasional analysis, while indexing and retrieval may become worthwhile when many questions reuse a large corpus. The comparison should include index construction and maintenance, as well as any available caching or pricing arrangements, because these can change the break-even point.

Design for changing models and documents

Larger context windows can make whole-document analysis feasible for more inputs, but capacity alone does not resolve positional sensitivity, evidence verification, or repeated-input cost [1][2]. Conversely, improved retrieval can make smaller contexts more useful, but it remains dependent on chunking, indexing, ranking, and source quality [3].

The durable architectural choice is therefore to make routing adjustable. Preserve document versions and source locations, reindex when content changes, enforce access controls before passages reach the model, and rerun evaluations when models or retrieval settings change. For sensitive material, apply the same permission rules to whole-document escalation as to retrieval; a fallback should not expose content the requester was not authorized to see.

The best route is the one that meets the task’s evidence and quality requirements within its operational budget. Use the whole document when broad relationships are essential and it can be processed reliably; use retrieval when questions can be supported by selected passages; and use escalation when the evidence available to the first route is insufficient.

References

[1] https://arxiv.org/abs/2307.03172 [2] https://arxiv.org/html/2407.16833 [3] https://www.anthropic.com/research/contextual-retrieval