Direct source inspection or Program-of-Thoughts?

Choose direct source inspection when the answer requires reading a small number of clearly labeled values. Choose Program-of-Thoughts (PoT) when the question requires a calculation substantial enough to benefit from executable code. In either case, the decisive control is the same: verify that every input comes from the right cell or chart series before accepting the answer.

Neither method makes provenance automatic. A direct answer can contain a plausible value read from the wrong row. A program can calculate perfectly using an invented input. The check must therefore connect each reported value to a location in the document and connect each operation to what the question actually requests.

Table note: AI Plaza is one observed operator in this workflow. GPT-6.1 Sol and Claude-Sonnet-5.5 are examples of models a reviewer could use to generate an answer for checking; neither model name is evidence that a particular answer is correct.

Check the source location before checking the arithmetic

A provenance audit asks where each input came from, not merely whether the final number sounds reasonable. For a table, the minimum evidence is the row and column for each value, together with the table title or page when the document has multiple tables. For a chart, the minimum is the named series and the relevant category, point, or interval.

The reviewer should then verify three separate propositions:

1. The value is present or defensibly estimated. It appears at the cited location, or the answer clearly identifies it as an estimate from a graphical scale.
2. The location answers the question. The row, column, series, period, and population match what was asked.
3. The operation is authorized. Any addition, subtraction, average, or comparison follows from the question and uses compatible inputs.

A correct calculation from the wrong cells fails the second test. A value copied accurately from the source can still fail the third if it is used in a total the question did not request.

A table example: correct arithmetic, wrong evidence

Consider an illustrative table with these entries:

• North, 2023: 120
• North, 2024: 140
• South, 2023: 80
• South, 2024: 90

If asked for North’s 2024 value, an answer of 140 should identify the North row and 2024 column. An answer of 210, calculated as 120 + 90, is arithmetically correct but unsupported: it combines different regions and years and does not answer the question.

If instead asked to add North and South for 2023, 200 is reproducible from 120 + 80. The table need not contain a printed “200” cell for that derived answer to be valid, provided the requested addition is appropriate and the categories can be combined. The answer should label 200 as calculated, not as a source-reported total.

This distinction prevents two opposite errors: accepting a fabricated source total and rejecting a legitimate calculation merely because the source did not print the result.

A chart example: the series is part of the value

For a line chart, “2024: 140” is incomplete if the figure contains several lines. The answer should identify, for example, the North series at the 2024 point, along with the unit and any relevant scale. For a stacked bar, the reviewer must distinguish a segment’s value from the height of the entire stack. For a pie chart, a slice percentage is not automatically an underlying count.

When a point falls between gridlines and no data label is present, report an estimate rather than an exact value. The source’s resolution and scale constrain precision; a model should not add decimal places that the figure cannot support. If labels, colors, or an interval boundary cannot be read reliably, the appropriate result may be “not determinable from this image,” rather than a forced number.

Reject totals the evidence does not support

A claimed total needs both inputs and an authorized aggregation rule. Reject it as unsupported when it is absent from the source and cannot be reproduced from the cited values under the operation requested by the question.

Check the following before accepting a sum:

• Coverage: Are all components of the requested total available, or are some categories missing?
• Overlap: Are the categories mutually exclusive, or could adding them double-count the same cases?
• Units and scale: Are values expressed in compatible units, currencies, and magnitudes?
• Time basis: Do the values cover compatible periods rather than overlapping or mismatched intervals?
• Hierarchy: Is a subtotal being added to its own components?
• Rounding: Could separately rounded inputs explain a small difference from a published total?
• Attribution: Is the result clearly identified as source-reported or reviewer-calculated?

Absence from the table alone does not prove a total is false. Conversely, being able to add two visible numbers does not prove their sum answers the question. The reviewer must establish both the arithmetic and its relevance.

If the source contains a printed total that differs from a calculation, preserve that distinction. Report the source’s total with its location and, if relevant, the independently calculated result with its inputs. Do not silently replace one with the other or assume rounding explains the difference without evidence.

Compare reasoning methods by the errors they can expose

Direct answers versus visible reasoning

Direct source inspection is efficient for a small lookup, but a bare answer such as “140” leaves the reviewer to reconstruct its provenance. Requiring a row and column, or series and point, adds a compact audit trail without requiring a long explanation.

CoT can expose the sequence of reasoning, but a plausible narrative is not proof that the model read the source correctly. The key evidence is still the cited input. A concise, verifiable location can be more useful to an auditor than a long account that never identifies the supporting cell.

Executable calculations versus grounded inputs

PoT is useful when arithmetic is the main source of complexity: it expresses a calculation as a program rather than relying only on prose. In the reported PoT experiments, the method improved on CoT by about 12% on average across the evaluated numerical-reasoning datasets. But among 198 PoT failures, 47% involved value-grounding errors and 33% involved logic errors. [1]

Those findings delimit what execution proves. A program can confirm that 120 + 80 evaluates to 200; it cannot by itself establish that 120 and 80 were the correct cells, that they refer to the same year, or that summing them was warranted. Audit the extracted inputs and the operation before treating the program output as an answer.

Compare chart extraction with reading the image

Chart-to-table extraction makes chart values easier to inspect and calculate over. It also introduces a separate question: does the extracted table faithfully represent the chart? In ChartQA experiments, supplying the gold underlying table brought models close to perfect results, while performance fell without it; extraction errors were especially consequential for less-common pie charts. [2]

A reviewer should therefore compare the extracted table back to the image, including the legend, axis labels, units, and category order. A clean-looking intermediate table is not a substitute for this check.

Broader evaluation reinforces the need for caution. ChartQAPro contains 1,341 charts from 157 sources and 1,948 questions, including unanswerable questions. The reported score for Claude Sonnet 3.5 was 55.81% on ChartQAPro, compared with 90.50% on ChartQA. The study identifies visual perception, instruction-following, and mathematical reasoning among its error categories. [3] The benchmark gap is not a measurement of performance on every document, but it is a reason not to infer robust chart reading from success on simpler examples.

Use a repeatable answer-checking workflow

1. Preserve the document context

Record the document, page or figure, table title, and relevant footnotes. Preserve merged headers and row hierarchy when extracting a table: a cell reference is only meaningful if its surrounding structure remains intact. Spreadsheet question answering research identifies preservation and interpretation of structure as a challenge. [4]

2. Request evidence before the result

For each input, ask the model to provide:

• Table: row label, column label, displayed value, and unit.
• Chart: series, category or interval, displayed or estimated value, and unit.
• Location: page, table, or figure identifier where available.

Then ask for the calculation separately. This format makes it possible to inspect the source read without accepting the arithmetic.

3. Independently inspect the cited locations

Open the original document rather than relying only on the model’s extracted text. Check whether the cited value is actually there and whether headers, legends, footnotes, and scale modifiers change its meaning. If a location cannot be verified, mark the input unresolved.

4. Recompute and classify the result

Recalculate using only verified inputs. Classify the answer as directly reported, derived from cited inputs, estimated from the chart, or not determinable. For a derived result, show the operation and identify any assumptions. A total that lacks sufficient components, combines incompatible values, or exceeds the requested operation remains unsupported.

The standard is not that every answer must contain a long explanation. It is that a reviewer can trace every material value to the source and reproduce every claimed operation.

References

[1] https://ar5iv.labs.arxiv.org/html/2211.12588 [2] https://aclanthology.org/2022.findings-acl.177.pdf [3] https://arxiv.org/html/2504.05506v1 [4] https://orbilu.uni.lu/bitstream/10993/67962/1/Spreadsheet_Survey.pdf