Experiment with 400 adults finds a Socratic AI tutor teaches more than explaining the mistake
Carnegie Mellon researchers compare four ways of giving feedback after a mistake and suggest that what works is sending the student back to the task until they get it right.
Michael W. Asher, Meng Cao and Paulo F. Carvalho, of Carnegie Mellon University's Human-Computer Interaction Institute, published a study on arXiv on 15 November 2026 that tests a widely held intuition: that a language model helps students learn more the better it explains why they went wrong. It is a research paper, with a preregistered experiment and open data and code, not a guide or an official recommendation.
The starting point is that the AI tutors evaluated so far have produced modest results. The authors recall the two-year trial of Khanmigo in 18 secondary schools, by Philip Oreopoulos and Nicole Low, in which the effect on standardised tests was small and the median student talked to the tutor in only 17% of the sessions in which they made mistakes. Their hypothesis is that a chatbot that waits to be asked comes too late: people who fail tend to give up. They propose a proactive tutor, built into the cycle of practice and correction.
The experiment brought together 400 adults in the United States who did not know how to set up constraint equations (for example, the one relating the metres of fencing available to the dimensions of an enclosure). After each attempt at three problems, they were randomly assigned one of four forms of feedback: seeing only the correct answer; a generic, pre-written explanation; that explanation plus a personalised one about their own mistake, generated on the spot by Gemini 3.5 Flash; or a conversation with a tutor, the same model, which was forbidden from giving the answer and had to guide them with Socratic questions.
The results favour the Socratic tutor. It led participants to correct 131 of their 206 mistakes themselves (64%), and they scored 9 percentage points higher (0.24 standard deviations) than the groups given explanations on three final questions similar to those they had practised. They also said the session had been more interesting, less frustrating and less costly, although it lasted almost twice as long: 13.3 minutes on average against between 6.8 and 7.3 in the other conditions. Personalised explanations did not improve on generic ones, and neither reliably outperformed simply showing the correct answer.
It should be read with its limits, which the authors acknowledge. The tutor's advantage appeared only on problems like those practised, not on the question about an untaught objective or on the problem that combined several, and the difference in the final test has a p-value of 0.048, right on the edge of significance. It was a single session of about twenty minutes, on one maths topic and with adults, not school students. Moreover, the Socratic condition bundled several things together (trying again, tailored questions and an encouraging tone), so the design does not separate which of them produces the effect. Nor does it compare with the same time spent on more exercises, and the cost, about 12 US cents per participant, grows with the number of users.
From this they draw a framework they call mastery experience: practice comes before explanation, the difficulty should produce informative errors without overwhelming, and the student needs to be able to retry until they succeed. Their reading is that the value of an AI tutor lies not in explaining the mistake more precisely but in turning it into a success of the student's own.
For teachers, the study does not prove any classroom practice, but it leaves design clues that can be tried out: asking the assistant not to give the solution and to break the problem down into smaller questions, and requiring a first attempt before any help. For context, the paper carried Stanford's review of 52 causal studies on AI in schools on 18 November, which reports uneven results. To get started, it is enough to read the abstract and sections 4 and 5.2 of the paper.
Experiment with 400 adults finds a Socratic AI tutor teaches more than explaining the mistake
Carnegie Mellon researchers compare four ways of giving feedback after a mistake and suggest that what works is sending the student back to the task until they get it right.
Michael W. Asher, Meng Cao and Paulo F. Carvalho, of Carnegie Mellon University's Human-Computer Interaction Institute, published a study on arXiv on 15 November 2026 that tests a widely held intuition: that a language model helps students learn more the better it explains why they went wrong. It is a research paper, with a preregistered experiment and open data and code, not a guide or an official recommendation.
The starting point is that the AI tutors evaluated so far have produced modest results. The authors recall the two-year trial of Khanmigo in 18 secondary schools, by Philip Oreopoulos and Nicole Low, in which the effect on standardised tests was small and the median student talked to the tutor in only 17% of the sessions in which they made mistakes. Their hypothesis is that a chatbot that waits to be asked comes too late: people who fail tend to give up. They propose a proactive tutor, built into the cycle of practice and correction.
The experiment brought together 400 adults in the United States who did not know how to set up constraint equations (for example, the one relating the metres of fencing available to the dimensions of an enclosure). After each attempt at three problems, they were randomly assigned one of four forms of feedback: seeing only the correct answer; a generic, pre-written explanation; that explanation plus a personalised one about their own mistake, generated on the spot by Gemini 3.5 Flash; or a conversation with a tutor, the same model, which was forbidden from giving the answer and had to guide them with Socratic questions.
The results favour the Socratic tutor. It led participants to correct 131 of their 206 mistakes themselves (64%), and they scored 9 percentage points higher (0.24 standard deviations) than the groups given explanations on three final questions similar to those they had practised. They also said the session had been more interesting, less frustrating and less costly, although it lasted almost twice as long: 13.3 minutes on average against between 6.8 and 7.3 in the other conditions. Personalised explanations did not improve on generic ones, and neither reliably outperformed simply showing the correct answer.
It should be read with its limits, which the authors acknowledge. The tutor's advantage appeared only on problems like those practised, not on the question about an untaught objective or on the problem that combined several, and the difference in the final test has a p-value of 0.048, right on the edge of significance. It was a single session of about twenty minutes, on one maths topic and with adults, not school students. Moreover, the Socratic condition bundled several things together (trying again, tailored questions and an encouraging tone), so the design does not separate which of them produces the effect. Nor does it compare with the same time spent on more exercises, and the cost, about 12 US cents per participant, grows with the number of users.
From this they draw a framework they call mastery experience: practice comes before explanation, the difficulty should produce informative errors without overwhelming, and the student needs to be able to retry until they succeed. Their reading is that the value of an AI tutor lies not in explaining the mistake more precisely but in turning it into a success of the student's own.
For teachers, the study does not prove any classroom practice, but it leaves design clues that can be tried out: asking the assistant not to give the solution and to break the problem down into smaller questions, and requiring a first attempt before any help. For context, the paper carried Stanford's review of 52 causal studies on AI in schools on 18 November, which reports uneven results. To get started, it is enough to read the abstract and sections 4 and 5.2 of the paper.