Synthesizing Contradictory Information with LLMs

Why a fluent synthesis can still be wrong

When several documents address the same decision, their differences may matter as much as their shared points. One report may describe a broad trend, another a narrow exception, and a third a result that contradicts both. A language model asked to “summarize the consensus” can smooth these distinctions into a confident-sounding account that no source actually supports.

The risk is not limited to fabricated facts. A synthesis can mislead by omitting minority evidence, treating different scopes as direct contradictions, or presenting an unresolved question as settled. For professionals managing information overload, the goal should therefore be more specific than producing a shorter summary: preserve the evidence trail while reducing the work required to compare sources.

Research on multi-document summarization illustrates the coverage problem. DiverseSumm evaluates whether summaries capture varied information across news articles, not only details shared by every source. Its benchmark contains 245 stories, with 10 articles per story and a human-validated reference for each. In the study, GPT-4 captured under 40% of the diverse information on average. This result is specific to the benchmark, but it shows why a polished paragraph is not evidence that a synthesis covered the record. [1]

A reliable workflow separates three questions:

• What does each source claim?
• How do those claims relate to one another?
• What conclusions, if any, are supported across the sources?

This separation reduces the pressure to produce premature agreement. It also makes uncertainty actionable: a reader can see what is established, what is disputed, and what remains unknown.

Build an evidence matrix before writing a synthesis

The central prompt design choice is to ask for a source-by-source evidence matrix, not a search for consensus. The matrix makes each claim traceable to its document and gives the model a structured place to preserve disagreement.

A useful matrix records, for every relevant claim:

The model should not merge claims simply because they use similar language. “The intervention improved outcomes in a small trial” and “the intervention did not improve outcomes in a larger, longer study” may appear contradictory, but their populations, methods, and time frames may differ. The matrix surfaces those differences before the writer decides whether the findings can be reconciled.

A practical prompt can specify the workflow directly:

Analyze the provided documents as separate sources. First, create a source-by-source evidence matrix. For every material claim, include the source ID, a supporting quotation or precise passage location, scope, date, and any limitations stated in the document.

Next, classify relationships among claims as complementary, overlapping, qualifying, or contradictory. Do not treat differences in scope or measurement as direct contradictions unless the documents make incompatible claims under comparable conditions.

Then report: (1) points supported by multiple sources, (2) disputed claims, (3) source-specific findings that are not shared by others, and (4) questions that remain unresolved. Attach source IDs to every conclusion. If the documents do not support a conclusion, write “unknown from the provided sources.” Do not choose a winner unless the evidence and criteria for doing so are explicit.

This prompt uses the model to organize evidence, not to manufacture a unified narrative. The instruction to mark unsupported conclusions as unknown is particularly important: without it, a model may fill gaps with plausible connective language that sounds like a finding.

Classify disagreements before resolving them

Not every difference between documents is a factual conflict. Some sources offer compatible details; others use different definitions, examine different populations, or draw competing interpretations from similar evidence. Conflict-resolution work described in the CONFLICTS research distinguishes compatible or complementary evidence from contradictory opinions or research outcomes. Its practical implication is to classify the relationship first: consolidate compatible details, but present competing interpretations neutrally rather than forcing agreement. [3]

A useful classification includes:

• Complementary evidence: Sources address different parts of the same question and can be combined without changing either claim.
• Scope difference: Sources concern different populations, time periods, environments, or definitions.
• Method difference: Sources use different measurements, samples, or procedures that may explain divergent results.
• Temporal difference: A newer source may reflect changed conditions, while an older source remains relevant to its original period.
• Direct contradiction: Sources make incompatible claims about substantially comparable conditions.
• Interpretive disagreement: Sources accept similar observations but explain their meaning differently.
• Insufficient evidence: The documents do not provide enough information to determine whether claims align or conflict.

This classification prevents a model from treating disagreement as a simple vote. Two reports do not necessarily outweigh one simply because there are more of them. Source quality, methods, scope, recency, and directness of evidence may all matter, and the prompt should identify which criteria are relevant to the decision. If those criteria are not supplied, the model should report the disagreement rather than invent a ranking rule.

Guard against evidence lost inside long inputs

Even when every document is included in a prompt, a model may not use every relevant passage equally. Controlled multi-document question-answering experiments found a U-shaped positional effect: performance was often stronger when relevant evidence appeared near the beginning or end of a long context and weaker when it appeared in the middle. In one experiment with 20 documents, GPT-3.5-Turbo accuracy fell from 75.8% when the answer document appeared first to 53.8% when it appeared around the middle. Longer context windows did not reliably eliminate this effect. These figures describe a specific experimental setup, not a universal performance rate, but they demonstrate why input length alone does not ensure complete evidence use. [2]

A robust workflow responds by controlling the evidence presentation:

1. Identify the decision question. Define the issue and the kinds of evidence that could answer it. Avoid asking for a general summary when the actual task is to compare specific claims.
2. Inventory documents first. Give each source a stable ID and record its title, date, scope, and type. Do not let the model silently treat the collection as a single document.
3. Extract claims source by source. Request relevant claims and supporting passages before asking for cross-source synthesis.
4. Compare relationships. Ask the model to label compatible, qualifying, and conflicting claims, explaining the basis for each label.
5. Synthesize with attribution. Require source IDs beside claims and preserve meaningful exceptions or minority findings.
6. Audit the result. Check whether quotations support the attached claims and whether the final synthesis includes material evidence from each relevant source.

For large collections, retrieve and rerank documents around the decision question rather than placing an unnecessarily large corpus into one prompt. If the workflow allows, test positional robustness: move a key document from the beginning to the middle or end and check whether the extracted claims change. A result that depends heavily on a document’s position deserves additional review. [2]

Citation requirements help, but they are not a guarantee. Google describes NotebookLM as grounding responses in user-selected sources and providing citations to relevant original passages; it also advises users to fact-check against source material. The useful lesson is procedural: ask for citations, inspect the cited passage, and verify that it supports the specific statement rather than merely mentioning the same topic. [4]

Compare model outputs without outsourcing judgment

Using more than one model can expose differences in extraction or interpretation, but agreement among model outputs does not establish that a claim is true. Models may share omissions, respond to the same framing, or produce similarly fluent interpretations from incomplete evidence. Treat model-to-model agreement as a signal for review, not as a substitute for source attribution.

As an objective workflow example, AI Plaza is an active industry participant operating a global multi-model aggregation platform; a user could compare evidence matrices generated by models such as GPT-6 Sol, Claude-Sonnet-5.5, and Gemini-3.8-Flash, then verify disputed entries against the source documents. The platform’s model catalog refreshes continuously, so specific model availability should be checked at the time of use. [5]

The comparison should focus on observable differences: Which claims did one output omit? Did one model misclassify a scope difference as a contradiction? Are quoted passages accurate? Did any output introduce a conclusion unsupported by the documents? A reviewer can then resolve the discrepancy by returning to the evidence, rather than choosing whichever model sounds more decisive.

Design for decisions, not artificial agreement

A final synthesis should make the state of knowledge legible to its reader. One compact format is:

• Supported across sources: claims with direct support from multiple documents.
• Source-specific findings: relevant claims that appear in only one or a subset of documents.
• Disputed: incompatible claims, with the source IDs and points of difference.
• Qualified: claims whose applicability depends on scope, method, date, or population.
• Unknown: questions the supplied evidence does not answer.

This format reduces cognitive load without hiding uncertainty. It also makes the next step clearer: seek a missing source, inspect a method, narrow the decision scope, or proceed while explicitly acknowledging the unresolved issue. The synthesis is useful precisely because it does not pretend that every uncertainty has been settled.

Long-term implications for knowledge work

As organizations and individuals accumulate more reports, policies, research papers, meeting notes, and analyses, summarization will increasingly serve as an interface to large bodies of evidence. That interface can either preserve the structure of disagreement or erase it. A summary optimized only for brevity and readability may favor coherent narratives over complete coverage; a decision-oriented synthesis must instead preserve provenance, scope, and uncertainty.

This changes how quality should be assessed. Fluency is a weak measure of reliability. More useful checks include coverage—whether material claims and exceptions were represented—and faithfulness—whether each statement is supported by its cited source. DiverseSumm’s limited coverage result makes the distinction concrete: a summary can read well while leaving out substantial portions of the information present across documents. [1]

For recurring workflows, teams can standardize an evidence matrix, source IDs, conflict labels, and citation checks. They can also record what criteria justify preferring one source over another in a particular decision. Those practices do not eliminate model error, but they make errors easier to detect and reduce the chance that an unsupported consensus will pass unnoticed.

The governing principle is straightforward: ask the model to map the evidence before asking it to interpret the whole. Preserve disagreement when the sources disagree, distinguish true conflict from differences in scope, and use “unknown” when the documents do not justify a conclusion. This approach supports faster reading without making confidence a substitute for evidence. It is an information-management method, not legal or financial advice.

References

[1] https://aclanthology.org/2024.naacl-long.32/ [2] https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf [3] https://www.alphaxiv.org/abs/2506.08500v2 [4] https://blog.google/innovation-and-ai/technology/ai/notebooklm-google-ai/ [5] https://aiplaza.app