A Screenshot Is an Incomplete Task Specification
A screenshot shows a rendered state at one moment. It does not, by itself, explain how that state was reached, what the user wants changed, or which action should come next. For UI and code-related work, this distinction matters: a model may identify a visible button or read a displayed value while still lacking the in
A rendered state is not an instruction
A screenshot shows a rendered state at one moment. It does not, by itself, explain how that state was reached, what the user wants changed, or which action should come next. For UI and code-related work, this distinction matters: a model may identify a visible button or read a displayed value while still lacking the information needed to act correctly.
A useful task specification therefore separates three kinds of evidence:
| Component | What it can establish | What it cannot establish on its own |
|---|---|---|
| Screenshot | Visible text, controls, spatial relationships, and displayed state | Prior interactions, hidden content, intended outcome, or whether a control is safe to activate |
| Task instruction | The requested outcome, target, and constraints | Whether the visible interface actually contains the described target |
| Interaction history | Relevant steps that led to the current state | Unseen system state or intent that was never recorded |
The distinction is not merely a prompting preference. It is central to visual instruction following: recognizing an element is different from grounding an instruction to that element, and both are different from inferring the user’s goal. A screenshot can be strong evidence about what is on screen while remaining incomplete evidence about what should happen next.
What Visual Systems Can Infer—and What They Cannot
Grounding an instruction is separate from describing the image
Modern vision-language systems can interpret screenshots as more than collections of pixels. Depending on the model and task, they may read text, identify interface elements, locate controls, interpret charts, or answer questions about a document. Yet the result depends on how the task is framed and what information is available.
The SeeClick work describes GUI interaction as grounding a language instruction in a screenshot. Its agent receives the current interface image, an instruction, and the previous four actions before predicting the next action. This setup makes an important boundary explicit: the image provides the current visual evidence, while the instruction and recent action history supply information that the image does not contain [1].
SeeClick’s ScreenSpot benchmark includes more than 600 screenshots and 1,200 instructions across iOS, Android, macOS, Windows, and web pages. Tasks involve locating text, icons, and widgets. In practical terms, the system must determine which on-screen element corresponds to a request, not simply produce a description of the screenshot [1]. If a person asks, “Open the export settings,” the model needs to identify the relevant control. If the person only uploads an image, the intended operation is unspecified.
Professional software raises the difficulty. ScreenSpot-Pro comprises 1,581 instruction–screenshot pairs from 23 applications, across five industry categories and three operating systems. Its tasks likewise require grounding natural-language instructions to precise locations. The evaluation focuses on dense, high-resolution software interfaces, where neighboring controls may be small and visually similar [2]. Performance on a simple page should therefore not be assumed to transfer to a complex production interface.
These findings point to three separate questions that should not be collapsed into one:
- What is visible? The system identifies text, elements, values, and their positions.
- What does the user want? The instruction states an outcome, target, or question.
- What is the appropriate next step? The system uses the instruction, visible state, and relevant history to propose an action.
A screenshot can help answer the first question. It may contribute to the other two, but it cannot reliably replace them.
How to Specify a Screenshot-Based Task
Name the outcome apart from the pixels
A practical specification should name the intended action separately from the pixels. For UI work, that means describing the outcome, identifying the target when possible, and distinguishing a request for analysis from a request to interact.
For example, “Here is the settings page” gives a model an image but no task. “Find the control for exporting a CSV; do not click it” provides a target and an action boundary. “Change the date range to the previous quarter, then tell me which control you would use before making any change” adds an outcome and a verification step.
When interaction history is available, include only the steps that help explain the present state. A useful description might say: “I opened the report menu, selected ‘Custom,’ and the date dialog is now visible. Identify the start-date field; do not submit the form.” This makes clear both what led to the visible state and what is being requested now. It also reduces the risk that the model interprets an already-open dialog as an instruction to continue an unknown workflow.
The same discipline applies to code shown in a screenshot. A code image may reveal a visible line, error message, or editor state, but it does not necessarily include the surrounding file, runtime context, preceding edits, or the user’s desired outcome. Ask for a specific operation—such as explaining a visible error, locating a function, or comparing two displayed snippets—and provide source text or execution context when the answer depends on details that pixels may obscure.
For professional use, specify:
- The question or requested outcome: What should the model determine or change?
- The target: Which page, control, chart, table, or region matters?
- The action boundary: Is the model being asked to describe, recommend, or perform an action?
- Relevant history: What preceding interaction explains the current screen?
- Evidence constraints: Should the answer use only visible content, or may it use supplied files or structured data?
- Verification needs: Should the model report a location, quote visible text, or wait for confirmation before acting?
This is not a guarantee of correctness. It makes the task testable: a reviewer can check whether the model selected the requested target and respected the stated action boundary.
Why Layout and Structure Matter Beyond Screenshots
Spatial grouping can change what a label means
Screens and documents contain spatial information that can be damaged when converted into plain text. Reading order, grouping, alignment, table boundaries, and proximity can determine meaning. A label may belong to the value beside it rather than the text above it; a chart legend may map a color to a series; a button’s location may distinguish it from a similarly labeled neighboring control.
ScreenAI illustrates one approach to preserving such information. The work describes a 5-billion-parameter vision-language model based on PaLI and flexible patching in the style of Pix2Struct. Its training includes screen annotation—identifying UI element types and locations—as well as question answering, navigation, and summarization. The authors report evaluations across web, document, chart, and infographic tasks, and introduce datasets for screen annotation and screen question answering [3]. The underlying lesson is that screen understanding involves element identity and spatial relationships, not just text recognition.
Charts make the issue especially clear. ChartQA includes 9,600 human-written questions and 23,100 generated questions that require chart reading and logical reasoning. The described approach combines visual features with chart data extracted into a table, using ChartOCR as part of that process [4]. A chart image may show the trend, but answering a precise question about values or comparisons can require recovering the underlying data. If the values are available in a table or source file, supplying them alongside the image can reduce ambiguity.
For tables and documents, preserve structure where possible. A screenshot can be appropriate when the question concerns layout or visible presentation. For exact calculations, cell-level comparison, or text extraction, a machine-readable table or source document is usually better evidence than a compressed image alone. For a chart, pair the image with the data when exact values matter. For a UI, include a screenshot at a resolution that keeps the relevant controls legible, and state the target in words.
As a workflow example, AI Plaza can be the place where the same screenshot and the same written task are compared across models. The comparison checks whether each model names the same target and states its location. It does not show that any model can safely perform the action. Disagreements should be checked against the screenshot and the task, not treated as proof.
What Reliable Interpretation Requires Over Time
Hold the image still and vary the instruction
As multimodal systems become more capable, the central operational question will not be whether a model can describe a screenshot. It will be whether the system can maintain a faithful distinction between observation, instruction, and action across a sequence of changing states.
That distinction has consequences for interface design and evaluation. A static screenshot tests perception and grounding. An interaction sequence also tests whether the system can use prior actions to interpret the current screen. A robust evaluation should therefore vary the instruction while holding the image constant, and vary the preceding interaction history while holding the final image and instruction constant. If the predicted action changes for the wrong reason, the system may be relying on superficial visual cues or ignoring relevant history.
A second implication concerns evidence preservation. Converting a page into a text-only representation can discard geometry; reducing a chart to an image can obscure precise values; cropping a screenshot can remove context that distinguishes nearby controls. Systems and workflows should preserve whichever form of evidence the task depends on: pixels for appearance, coordinates for location, text for exact wording, structured data for calculation, and action history for sequence.
Finally, the specification should make uncertainty visible. If a model cannot distinguish two adjacent controls or cannot infer whether a dialog is in a saved or unsaved state, it should be able to report that limitation rather than silently guess. The user can then provide a closer crop, clarify the desired outcome, supply the previous action, or request a description instead of an interaction.
A screenshot is useful because it records visible evidence. It is incomplete because it does not encode the full task. Reliable work comes from treating the image as one part of a specification: pair it with a named question or intended action, include relevant interaction history, and retain the layout or structured data needed to support the answer.
References
[1] https://arxiv.org/html/2401.10935v2 [2] https://arxiv.org/html/2504.07981v1 [3] https://arxiv.org/pdf/2402.04615 [4] https://arxiv.org/html/2203.10244