A selective teacher helps AI learn new information with less training
The authors report similar question-answering accuracy with 38–69% fewer supervised training tokens across three research tasks.
Published November 13, 2026Briefs / Artificial Intelligence / Generative Models
Researchers at Stanford and ETH Zürich have tested a way to teach a language model new information while giving it more room to work through an answer itself. Their method, Speculative Self-Distillation, brings in a teacher’s next word or word fragment when the teacher and student, derived from the same starting model, disagree sufficiently.
The aim is to put knowledge into the model so it can answer later without reopening the material. During training, the teacher receives a reference answer to each question. The student learns to answer with less help, with the teacher stepping in at points where their predictions diverge.
In a paper posted November 9, the researchers report reaching similar question-answering accuracy with 38–69% fewer supervised tokens—the pieces of response text used to train the model—than a comparison method in which the student writes every response and the teacher supplies feedback. Across the three tasks, accuracy stayed within 1.4 percentage points of that comparison. The tasks covered a fictional company’s documents, recent information from Wikipedia and undergraduate chemistry questions.
The savings count all response positions used for training across successive updates. The teacher still evaluates every position, and every position contributes to the learning signal. Selecting when the teacher writes helps the student reach the measured performance with fewer such training tokens. It also requires the system to run the teacher alongside the student; the token result alone gives no measure of energy or financial savings.
A separate chemistry experiment, using a different base model, tested whether learning the new subject damaged earlier capabilities. Across six general benchmarks, the selectively guided model’s average score fell by about 1.4 percentage points, compared with about 4.5 points for the student-written comparison. Some forgetting remained. The published work and released code establish a research method for short-answer learning; its performance on long explanations and complex reasoning remains an open question.