What reinforcement learning adds to a vision-language model

Highlights

1The claim

RL with a verifiable reward trains a model on its own samples and pays it only when a program confirms the answer. For language models this step turns a fine-tuned model into a reasoning one. Vision-language models answer a different mix of questions: many are settled by looking rather than deriving, and a reward that checks only the final answer can just as well teach a model to stop thinking. The comparisons that would show when RL helps are rarely run under one recipe, one evaluation and matched data. We run them.

Whether RL beats SFT depends on what the task pays for. RL mostly sharpens what the model can already sample; where the answer is out of reach, SFT has to go first. The rest of this report hangs from that claim, and each branch links to the page with all of its evidence.

• Reasoning pays: RL wins and keeps the reasoning. evidence
• Reasoning doesn't pay: RL changes form, not ability. evidence
• Knowledge missing: SFT first, RL polishes. evidence
• What RL's gain is made of: it needs the image and moves the model less. evidence
• Why trust it: the harness every number passes through. evidence
• Open: one model for photos and charts. evidence

2Setup

All models start from Qwen3-VL-2B-Instruct or Qwen3-VL-8B-Instruct and train LoRA adapters of rank 16 on the language model's attention and MLP layers in bf16; the vision encoder stays frozen. Three methods are compared on the same training questions.

Rollouts come from vLLM serving the trained adapter unmerged, so the sampler is the policy being trained: in a controlled check the sampler–trainer gap falls from 0.132 to 0.008 nats per token. One RL epoch draws each training question once, the exposure SFT has.

Each candidate dataset passes a screen on 500 training questions before any training, with gates written before the candidates ran: a black or swapped image must cost at least 10 points, at least 25% of questions must give GRPO a gradient, answers must be parseable, and accuracy must not already sit at the ceiling (Table A1).

3Where reasoning pays, RL wins and keeps it

Trained with step-by-step rollouts for one epoch on ChartQA's human-written questions, GRPO raises accuracy from 69.7% to 80.1% (+10.4 [+8.2, +12.6] points) and keeps its reasoning (205 → 162 tokens). SFT on the same questions reaches 68.8% at best, so GRPO ends 11.3 points above it. The audited judge credits many of Instruct's misses that are right answers in the wrong form, such as "20%" for a label of 20; under it the gain is 5.4 points (77.2% → 82.6%).

The answers were already within reach. Sampled 32 times per question, GRPO raises pass@1 from 66.4% to 79.8% and ends level with Instruct at pass@32 (91.0% → 91.5%): it learns to choose what Instruct could already sample. With short-answer rollouts on the same data, it gains at pass@1 and gives up coverage at pass@32 (86.1% → 77.1%).

Scope: ChartQA is the one chart source in this study, and SFT in this comparison trains on its short reference answers. A second chart-and-document source and SFT on reasoning traces are the next tests.

4Where it doesn't, RL changes form, not ability

A 0/1 correctness reward keeps whatever raises the chance of a correct final answer. On photo questions, reasoning lowers it. On VQAv2 under the reasoning prompt, GRPO goes from 55.1% at 148 tokens to 78.3% at 8 tokens, the level of Instruct's short answers (77.9%). The harness separates the two readings: the exact-match gain of 23.2 points is 4.4 under the audited judge.

Short-prompt RL meets SFT on this data rather than passing it, 78.6% against 79.9% on VQAv2, and it narrows what the model can sample: pass@32 falls from 79.8% to 76.6% while SFT raises it to 82.4%.

5Where the answer is out of reach, SFT goes first

On medical images GRPO started from SFT is the best model in all three settings, and SFT beats RL from Instruct in domain. SFT teaches the domain's answers; RL then polishes them, adding 1.3 points on SLAKE at 2B.

Reach separates the two methods. On SLAKE's open-answer questions Instruct's pass@8, the share of questions where one of GRPO's eight samples is right, is 53.7%. GRPO ends close to it at 56.1% and SFT passes it at 68.6%; 8B repeats this (56.7, 59.5, 72.1). Across 10 open-answer rows RL from Instruct ends at most 2.8 points above reach, and only on SLAKE does SFT pass it. The proposed mechanism, in the spirit of what Yue et al. (2025) report for text, is that RL can only reward answers the model samples while SFT can teach the others; the pattern is measured here per domain, and the mechanism itself is untested.

6What RL's gain is made of

Replacing each image with a black one shows what a model answers without looking. RL from Instruct answers as many questions blind as Instruct in 5 of 6 settings, every interval including zero, and +3.2 points in the other. SFT answers more without the image in every setting it trained in. At 8B, 10.4 of SFT's 19.1-point gain on SLAKE survives a black image, against 0.4 of RL's 10.3. RL improves the answers that depend on the image; SFT also learns which answers are likely for a question.

On medical data RL also moves the model far less: its forward KL from Instruct is 5 to 16 times smaller than SFT's. RL's Razor (Shenfeld et al., 2025) finds that forgetting tracks this distance, and GRPO loses less than SFT on PathVQA in every setting. On photos and short-answer charts, where SFT's labels are close to what Instruct already says, both distances are small and SFT's is the smaller.

On sets no model trained on, RL keeps a small edge over SFT: +0.9 [+0.2, +1.6] points on TextVQA and +1.0 [+0.5, +1.6] on VizWiz. SFT on exactly the 4,000 questions GRPO drew does not close it.

These measurements describe RL from Instruct; GRPO after SFT keeps SFT's priors, with a black-image score +24.0 points from Instruct's on OmniMedVQA's held-out collections. In short-answer settings RL is also more confident than it is accurate, with a calibration error of 12.1–23.7 points against SFT's 2.2–8.6.

7Why trust it: the harness

Nine layers, each answering one question about a result, each with its own measured number. Every difference reported here passes through all of them.

Every table is regenerated from 1,268 saved evaluation records, and a test fails when a published table or a quoted number drifts from them. The judge and the audit cost $6.21 in API calls.

8Open: one model for photos and charts

Does one model learn when to reason? Trained on photos and charts together, a 2B model kept its chart reasoning until update 1,750, lost it by update 2,000, and ended at 65.9% on ChartQA against 80.1% for the chart-only model. The cause is mixing, not training length: a chart-only model trained for the same 2,844 updates keeps 179-token answers and its dev accuracy (75.8% → 75.8%). Routing prompts by domain is the next test.

9Next

1. A strategy-by-category map. The same arms on a table grouped into general VQA, reasoning, document and chart, perception and hallucination benchmarks, with an image-gain column for each, plus one new training source each for reasoning and for documents. Predictions for the new sources are written before they train.
2. Error analysis that names the errors. Rule-based checks on every evaluation and labeled samples at milestones, so each error type can be taken to the literature on reducing it.
3. SFT on reasoning traces. The direct test of whether RL's chart gain comes from RL or from its reasoning targets.
4. Medical imaging as its own milestone, from 2D medical questions to 3D abdominal CT.

Side note: do 3D CT benchmarks need the scan?

The same harness asks how much of two CT benchmarks is reachable without looking. On DeepTumorVQA a text-only rule scores 60.7% against 34.5% chance, and M3D-VQA's correct option is A in nearly half of its questions. A debiased DeepTumorVQA built here brings the rule to 35.2%, and on it GRPO and SFT reach the same accuracy. Full note.