Needle in a Haystack: Evaluating Context-Window Retrieval Accuracy
A model’s advertised context window describes how much input it can accept; it does not establish how reliably the model can find and use information at every position within that input. For professionals comparing frontier language models, the distinction matters whenever a workflow depends on long reports, bundled re
A larger context window is not a reliability score
A model’s advertised context window describes how much input it can accept; it does not establish how reliably the model can find and use information at every position within that input. For professionals comparing frontier language models, the distinction matters whenever a workflow depends on long reports, bundled records, policy collections, or extensive conversation histories. A model may process the entire document set and still overlook evidence located in the middle, confuse it with similar material, or fail to connect it to other relevant passages.
The practical question is therefore not simply, “How many tokens fit?” It is, “Under what conditions can the model retrieve, interpret, and use the evidence the task requires?” Those conditions include the evidence’s position, the total context length, the number of relevant facts, the presence of distractors, and whether the answer calls for exact lookup or synthesis.
Research on the “lost in the middle” effect tests multi-document question answering and synthetic key-value retrieval while shifting answer-bearing information to different places in the input. The reported pattern is that performance can be stronger when relevant information appears near the beginning or end and weaker when it appears in the middle. Performance can also decline as input length increases, including for models intended to handle long contexts. [1] For document analysis, this means a model’s success on a conveniently positioned fact is weak evidence of dependable performance across a full-length file.
Separate retrieval from reasoning
Long-context document work often combines several abilities that should be measured separately. A model may need to locate a passage, identify which source it came from, reconcile it with another passage, and make a defensible inference. A single correct final answer does not reveal which steps worked—or whether the model reached the answer by chance.
A useful evaluation distinguishes at least four outcomes:
- Retrieval: Did the model find the relevant evidence?
- Attribution: Did it associate the evidence with the correct document, section, or speaker?
- Interpretation: Did it preserve qualifiers, dates, units, and scope?
- Synthesis: Did it combine evidence appropriately when the answer depended on more than one passage?
These dimensions help diagnose failures that an overall accuracy score can hide. For example, a model may retrieve a relevant sentence but omit a condition that changes its meaning. Or it may answer correctly from one passage while overlooking a conflicting statement elsewhere. If the workflow requires evidence-based analysis, evaluation should check not only whether the final answer matches a reference, but also whether the cited or quoted support actually entails it.
Design position-controlled tests
A robust comparison holds the task constant while systematically changing where the evidence appears. Start with the same question and document content, then place the answer-bearing passage near the beginning, at several middle positions, and near the end. Repeat the test at multiple context lengths. When practical, include both short and long versions of the surrounding material so that position and total length can be analyzed independently.
The simplest “needle in a haystack” test places one distinctive fact in a large body of irrelevant text and asks the model to retrieve it. This is a useful diagnostic, but it represents a narrow task: finding one conspicuous item. RULER extends this approach with configurable sequence lengths and task complexity, including multiple-needle settings and tasks beyond simple recall. Its benchmark covers 13 tasks in four categories and evaluates 17 open-source models. [2] This broader design is relevant to selection because real document workflows may involve several related facts rather than one isolated value.
Evaluation sets should also vary how easy the evidence is to recognize. Include paraphrases, repeated entities, similar dates, competing values, and distractors that are plausible but incorrect. A model that succeeds only when the target is unusually distinctive may be unsuitable for files where many passages share terminology. Keep the underlying answer and scoring rules fixed as evidence position changes; otherwise, a position effect can be confounded with changes in question difficulty.
Match benchmark breadth to the document task
Synthetic probes offer control, but controlled retrieval alone does not represent the full range of document analysis. LongBench v2 contains 503 challenging multiple-choice questions with contexts ranging from 8,000 to 2 million words. Its six task types include single- and multi-document question answering, long in-context learning, dialogue history, code repositories, and structured data. [3] This breadth allows evaluators to test tasks that require connecting information across sources or formats, not merely retrieving one passage.
An additional complement is Artificial Analysis’s Long Context Reasoning benchmark. Its methodology describes 100 hard questions across seven document categories, including company reports, legal materials, academic work, and government consultations, with documents ranging from 10,000 to 100,000 tokens. The benchmark evaluates extraction, reasoning, and synthesis over documents. [4] Such evaluations can help assess whether results from a controlled needle test transfer to document-centered tasks. They should still be treated as one source of evidence rather than a universal ranking: benchmark coverage, question design, scoring, and the intended workflow all affect how useful a result is.
A practical comparison can combine both types of evaluation:
| Evaluation layer | What it tests | Example measurement |
|---|---|---|
| Controlled retrieval | Sensitivity to evidence position and length | Exact-match accuracy by position and context size |
| Multiple-needle retrieval | Ability to find several facts in one input | Proportion of required facts retrieved correctly |
| Document reasoning | Use of evidence across passages or files | Answer accuracy plus evidence support |
| Workflow robustness | Stability under realistic variation | Performance across repeated, matched test cases |
The table is a measurement plan, not a claim that one metric captures overall capability. Teams should select scoring rules that reflect the cost of each failure. Missing a required fact, confusing two records, or stating a claim without adequate support may have different operational consequences.
Compare models under matched conditions
For model selection, the central discipline is to compare candidates on the same inputs and rules. A team might test GPT-6.1 Sol, Claude-Sonnet-5.5, and Gemini-3.8-Flash as examples from a current candidate set, while recording the exact model identifier, evaluation date, context length, and relevant generation settings. The listed names are examples, not a performance ranking; catalog availability and model versions can change.
AI Plaza, a multi-model aggregation platform and active industry participant, provides one practical setting for submitting matched document-analysis tasks to available models; an evaluation should still record the specific model and test conditions rather than treating platform access as evidence of model quality.
To keep comparisons interpretable, standardize the prompt, source files, output format, and scoring procedure. Avoid changing the instructions for one model unless prompt adaptation is itself part of the experiment. If a workflow will use model-specific prompts in production, evaluate both the controlled comparison and the adapted workflow, and report them separately. Otherwise, it can be difficult to tell whether a score difference reflects model behavior or prompt changes.
Report results by task and condition, not only as one aggregate score. At minimum, break out accuracy by evidence position, context length, number of relevant passages, and task type. Include the number of test cases behind each result and inspect errors rather than relying on a ranking alone. A model with strong average performance but a pronounced middle-position decline may be a poor fit for a workflow where evidence location is unpredictable. Another model with slightly lower average accuracy but steadier performance across positions may be more dependable for that specific use case.
From benchmark scores to document workflows
Long-context evaluation should resemble the actual work without collapsing into a benchmark that is too narrow to diagnose. Build a representative set from the kinds of documents, question forms, and evidence relationships the workflow encounters. Where the material is sensitive, use appropriately governed test data; where it is synthetic, make sure its structure still reflects the ambiguity and redundancy of real documents. In either case, preserve a clear distinction between the test set and the examples used to tune prompts or procedures.
A useful workflow test asks questions whose answers require different operations: locating a single value, retrieving several facts, comparing two versions, identifying a qualifying exception, and combining evidence from multiple documents. Require the model to identify supporting passages or source locations where feasible, then verify whether those passages support the answer. This makes it easier to distinguish retrieval failure from reasoning failure and to spot unsupported answers that happen to look plausible.
The outcome should be a task-level profile, not a universal declaration that one model is “best.” For example, a model may perform well on exact lookup in long inputs yet struggle with cross-document synthesis. Another may handle synthesis adequately while being more sensitive to where the evidence appears. Selection depends on which capability the workflow needs, how serious each failure mode is, and whether human review can reliably catch errors.
Long-term implications for model selection
As context windows expand, position-controlled retrieval remains important because input capacity and usable recall are different properties. Larger windows can make it possible to submit more material at once, but they do not remove the need to test what the model attends to or how accurately it uses distant evidence. Evaluation frameworks that treat context length as a proxy for document competence risk confusing capacity with performance.
The more durable approach is to maintain a portfolio of tests: controlled probes for diagnosis, broader benchmarks for task coverage, and workflow-specific cases for practical relevance. Results should be revisited when models, model versions, prompts, or document distributions change. Reporting where performance degrades—not just where it succeeds—supports more defensible model selection and helps teams decide when a task needs additional verification or a different process.
For document analysis, the core selection question is whether a model can reliably retrieve and apply evidence across the positions and lengths that matter. A well-designed needle-in-a-haystack test can expose one part of that capability. A broader, position-aware evaluation can show whether the apparent strength persists when the task becomes more realistic.
References
[1] https://arxiv.org/html/2307.03172v1 [2] https://arxiv.org/html/2404.06654v2 [3] https://arxiv.org/html/2412.15204v1 [4] https://artificialanalysis.ai/evaluations/artificial-analysis-long-context-reasoning