Projects

VQA-RLVR: what reinforcement learning adds to a vision-language model

Qwen3-VL at 2B and 8B, trained with SFT, with GRPO on a 0/1 correctness reward, and with SFT then GRPO, on photo, chart, and medical questions.  2026

Reinforcement learning with a verifiable reward is now the standard step after supervised fine-tuning for language models. For vision-language models it is less clear that it adds anything, because most visual questions are answered by looking rather than deriving. This project runs that comparison under one recipe and scores every model through one harness, built to tell a real gain from a change in answer format, and use of the image from answers learned without it.

Accuracy (%), Qwen3-VL-2B unless marked; base is the released Instruct model. ChartQA is answered step by step; SLAKE is medical; no model trained on TextVQA. The full matrix, with 95% intervals, is in the report.

RL wins where reasoning pays: on charts it keeps its reasoning and lands about ten points above every short-answer model. On medical images fine-tuning teaches the domain's answers, and RL started from the fine-tuned model is the best in every setting. RL's gains also need the image. With a black image in its place, the RL model answers no more questions than the base model in five of six settings, while fine-tuning on the same questions answers up to 15 points more. Pointed at two 3D CT benchmarks, the same harness found most of their published accuracy reachable without the scan; with those shortcuts removed, RL and fine-tuning tie.

Code Report

VLM-Rec: a multimodal recommender, and where generative retrieval loses recall

Retrieval to serving on Amazon Reviews 2023, with a stage-matched audit of semantic-ID generative retrieval.  2026

A full cascade in which every stage reads item content: frozen text and image encoders, plus a profile Qwen2.5-VL writes for each item. The same system then serves as a lab for the question a team adopting generative retrieval has to answer: which stage loses the recall?

The 208K-item Beauty catalog runs on the same code, reached by changing one config file.

The tokenizer loses the most recall and beam search comparatively little, so tokenizer work repays effort before wider beams do.

Code Technical report

A small vision-language model for clinical questions

SigLIP paired with SmolLM2, roughly 445M parameters, trained from scratch on 200k image and text pairs.  2025

Six variants, built to isolate what actually mattered rather than to report one number: all four corpora against one domain at a time, and LoRA against full fine-tuning. LoRA at rank 16 held 97% of full fine-tuning accuracy while training a third of the parameters. The lesson that transferred to later work came from the data mix. Without temperature-weighted sampling the largest corpus took 96.5% of every batch and drowned the specialist sets; at T = 0.5 that fell to 79%.

Fixing the labels instead of the model

Two data-curation pipelines with a foundation model in the loop.  2026

Our results had stopped improving, and the reason was not the architecture. It was that annotation is slow, and slow annotation means noisy labels. One pipeline re-segments thousands of preclinical MRI timepoints with MedSAM2 and cross-checks each result against geometric sanity tests, so a human reviewer only has to keep, accept, or edit. The other replaced whole-slide annotation across a five-stain pathology cohort.

Swapping the cleaned labels into an unchanged downstream model moved prognosis AUROC from 0.70 to 0.90, at about a fifth of the annotation time the lab had budgeted. It also surfaced the strongest signal in the dataset, an immune-cell ratio the previous labels had hidden completely, and caught a contaminated slide before it reached the cohort statistics.