What a 2B Vision Model Sees vs. What It Knows

What if a 2-billion parameter model could look at your screen, read an instruction like "close the window," and click the exact right pixel? That's the question behind Pixel Pilot — a research project probing the internal representations of Qwen3-VL-2B-Instruct to understand how a small vision-language model encodes spatial information about UI elements.

The Architecture

Qwen3-VL-2B has 28 transformer decoder layers with a hidden dimension of 2048. Its vision encoder (ViT) uses a patch size of 16 with spatial merge of 2. A 504px image produces roughly 90–225 visual tokens arranged in a grid — say 15×15. The token layout is: system tokens, then visual tokens, then instruction tokens, then generated tokens.

This ordering matters. Causal masking means visual tokens are processed before instruction tokens — the model builds a generic visual representation of the screenshot without knowing what it's looking for. It can't "look back" at the image conditioned on the task. This is the fundamental limitation we're working against.

Finding 1: Attention Heatmaps

The first approach: extract cross-attention weights from decoder layers 12–17, from the generated answer tokens back to the visual tokens. Aggregate across layers and heads to produce a 2D heatmap over the image. Apply softmax sharpening (temperature 0.05) and top-k masking (top 5%). The weighted centroid becomes the predicted click point.

# Extract attention heatmap from model internals
def extract_heatmap(model, image, instruction):
    # Forward pass with attention output
    out = model.generate(image, instruction,
        output_attentions=True, max_new_tokens=32)

    # Collect attention to visual tokens at layers 12-17
    attn = stack([out.attentions[l] for l in range(12, 18)])
    attn = attn[:, :, -gen_len:, vis_start:vis_end]

    # Average across layers, heads, generated tokens
    heatmap = attn.mean(dim=(0, 1, 2))  # → (n_visual_tokens,)

    # Reshape to spatial grid and sharpen
    grid = heatmap.reshape(rows, cols)
    grid = softmax(grid / 0.05)  # temperature sharpening
    grid[grid < topk_threshold(grid, k=0.05)] = 0

    return weighted_centroid(grid)  # → (x, y)

I evaluated this on the full ScreenSpot benchmark — 1,272 screenshot samples with labeled click targets. The results were humbling.

1.9% accuracy. Essentially random. The model's attention is too diffuse — it "looks" broadly across the entire screen rather than focusing on the instruction target. But 25% of predictions land within 200 pixels. The spatial signal exists in the attention maps, it's just noisy.

Finding 2: Hidden State Embeddings

Here's the key insight: the last token's hidden state — the position right before the model starts generating text — has seen the entire image AND the full instruction via causal self-attention. It's the only position in the sequence that's truly instruction-aware. If spatial knowledge exists anywhere in the model, it's here.

I extracted two things per sample: the visual hidden states (the 2048-dimensional embedding at each visual token position — per-patch image features) and the query hidden state (the 2048-dimensional embedding at the last token position — the instruction-aware representation). Each visual token maps to a known (x, y) region on screen.

First, I measured the spatial resolution ceiling: how close is the nearest visual token to the true click point? The answer: 54.3px average distance, with 45.3% of samples having a visual token within 50px of the target (576 out of 1,272). The grid itself has enough resolution — the problem is selecting the right token.

Finding 3: Decoding the Internals

If the model knows where things are but can't say so, can we train a small decoder to read its mind? I trained three lightweight neural networks on the extracted features.

RefinementNet takes the 2D attention heatmap (1×15×15) plus the centroid coordinates as input, runs it through a 2-layer CNN with adaptive pooling, and outputs a refined (x, y) prediction. This achieved the best validation loss of 0.044 — proving that even the noisy attention maps carry learnable spatial structure.

GroundingNet takes just the last-token hidden state — a single 2048-dimensional vector — and decodes it through an MLP (2048→512→128→2) with dropout. Validation loss: 0.067. A single vector, decoded by a tiny network, can predict click coordinates. The model does encode spatial information in its hidden states.

# GroundingNet — decode click location from hidden state
class GroundingNet(nn.Module):
    def __init__(self, hidden_dim=2048):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(hidden_dim, 512),
            nn.ReLU(), nn.Dropout(0.3),
            nn.Linear(512, 128),
            nn.ReLU(), nn.Dropout(0.2),
            nn.Linear(128, 2),
            nn.Sigmoid()  # output normalized (x, y)
        )

    def forward(self, query_hidden):
        return self.net(query_hidden)  # → (batch, 2)

CombinedNet fuses both signals — per-token attention values and PCA-compressed hidden states (2048→32 dimensions) — on the spatial grid. A per-cell MLP processes each token's features, then a 2D CNN (64→32→16 channels) captures spatial relationships, followed by an MLP head. Validation loss: 0.073.

What This Means

There's a stark gap between what the model sees and what it knows. Attention heatmaps — what the model "looks at" — give 1.9% accuracy. But the hidden states — what the model internally represents — encode enough spatial information for a tiny MLP to decode click targets with significantly lower error.

The bottleneck isn't perception. The 2B model perceives UI elements and encodes their locations. The bottleneck is expression — the model can't surface that spatial knowledge through its text generation interface. Causal masking means the visual representation is built generically, and the spatial precision gets lost in the attention-to-text pipeline.

What's Next

The decoder networks are proof-of-concept — tiny models trained on extracted features. The next step is attention-guided seeding: using the heatmap to bias an initial search region, then running the binary search refinement within that region. Beyond that, lightweight LoRA fine-tuning on a small set of click examples could close the gap between what the model knows and what it can express.

All of this runs locally on an M3 MacBook Air with 8GB RAM. No cloud, no fine-tuning for the core findings. The viewers above show every single one of the 1,272 evaluated samples — you can filter by pass/fail and see exactly where the model's attention lands versus where it should.

Swapnil Jadhav

Full Stack Developer · New York

Share