MIT researchers use Astra to repair a small Qwen model’s reasoning loops

Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT’s Computer Science and Artificial Intelligence Laboratory in Cambridge, Massachusetts, reported on November 2 that OpenAI’s GPT-6 Astra had reduced a recurring failure in Qwen3.5-0.8B, a language model with about 800 million parameters. Astra identified a source of repetitive reasoning and altered a small set of internal numerical weights. The team’s research report presents the result as an early example of a general-purpose AI inspecting and modifying another network.

The released evaluation records put MATH-500 accuracy at 48.1% after the strongest edit, up from 7.9% for the original model under the same sampling settings. GSM8K rose from 13.8% to 67.1%, and instruction-following accuracy on IFEval rose from 21.0% to 62.3%. On MATH-500, the share of responses exhausting the 32,768-token allowance fell from 93.0% to 33.1%. Those numbers describe controlled benchmark runs with repeated samples per problem.

The change largely recovered answers that repetitive thinking had prevented the model from finishing. Astra’s final experiment report describes a separate check on 1,152 held-out practice problems: 576 of 579 newly correct answers came from responses that the original model had truncated at its token limit. Accuracy across the originally finished responses was unchanged overall. This is evidence for a targeted reliability repair, rather than a general method for teaching a model new mathematics.

The repair targeted feed-forward components in layers 15 and 19. These components transform a model’s internal representation before passing information onward. Astra subtracted a direction associated with repetition from 64 output columns in each layer, touching 131,072 weight positions, about 0.016% of the model. The published edit manifest records 130,188 values that actually differ after rounding to the precision used for evaluation.

The process used examples and iterative experiments despite excluding gradient descent, the usual procedure for calculating weight updates from training errors. Astra could inspect practice problems and answers, run Python over the weights, and request model outputs or internal measurements. The successful run consumed 50 forward-pass requests, 80 Python runs and 158 agent turns. A single forward-pass request could cover hundreds of problems. The records therefore document a substantial diagnostic search, with practice data kept separate from the final benchmarks.

The team also tried the approach on Gemma 3 270M, whose errors more often involved arithmetic and algebra than repetitive output. Its follow-up records show MATH-500 scores moving from 6.8% to between 7.0% and 7.7%; the researchers describe a meaningful improvement as unestablished. For developers, that comparison makes the likely first use clearer: investigate a recognizable failure mode in an existing model, then test whether a small intervention repairs it.

A downloadable edited Qwen checkpoint makes the result available for further testing. Its model card reports a separate full MATH-500 rerun scoring 51.0%, while the accompanying base-model check covered only 25 problems. That provides additional evidence for the edited model’s performance, with a narrower comparison than a complete paired replication. Claims about production reliability would require testing the applications, prompts and sampling settings a developer actually intends to use.

The financial case also remains open. For a developer with suitable model weights and a well-defined defect, the relevant expense would be finding and validating a successful edit. The released run history includes repeated inference, agent reasoning and experimental checks. A useful commercial comparison would measure that complete bill against ordinary fine-tuning, generation-setting changes and application-level safeguards, including any regressions elsewhere.