The easiest number in any AI rollout report is how many people have tried the thing, and it is the least useful one to decide with. Today Rex has a two-year randomized trial that collected the harder one. A tutor sat one click away from struggling students for two school years, and the logs show how often they opened it at the moment they got something wrong. The vendor even changed the design halfway through, and the logs caught that too. What to carry out of a classroom study into a company is a short list, and Rex marks where it stops. — Muximus
Two school years, 18 middle schools, an AI tutor one click away during every daily math session for students who were behind. Ninety-six percent of them tried it. The median student used it in 17% of the sessions in which they made a mistake.
The learning gain was real, modest, and about what the same platform produces without the AI. The study is by Philip Oreopoulos (University of Toronto) and Nina Low (Charles River Associates), who believe it is "the largest and longest experimental evaluation of an AI tutoring program in public schools to date." It measures a gap that matters to any AI rollout: having access to an assistant, and using it when it would help.
The promise was serious, and the test was fair
The idea deserves its strongest form first. In 2023 Sal Khan described the goal as "giving every student on the planet an artificially intelligent but amazing personal tutor." The economics behind it are sound. Individualized tutoring is among the most effective interventions in education: high-dosage programs produce 0.2 to 0.4 standard deviations a year, at a cost of several thousand dollars per student. A language model drops the marginal cost of individual feedback close to zero.
This trial was built to remove the usual excuses. Hamilton County Schools in Tennessee ran it over the 2024-25 and 2025-26 school years. Within each school, one or two grades were randomly assigned to use Khan Academy with its AI tutor, Khanmigo: 53 grade-level clusters, 28 treated and 25 control. The program replaced the math content of the district's existing daily remedial sessions, 25 to 40 minutes in groups of four to eight. Khanmigo was set to coach, not answer - it withholds answers and asks guiding questions. Access sat inside mandatory, supervised time, not in an optional extra. The study is registered with the AEA RCT Registry (AEARCTR-0013519).
One design point governs everything after it. The control classes were not technology-free; many used other adaptive math software without a conversational AI. The authors say plainly that the comparison "should not be read as isolating the AI tutor."
By the numbers: the result
- +1.3 national percentile ranks per term.
- 0.06 to 0.08 standard deviations over a school year.
- 0.14 standard deviations implied for a full year of participation under mature implementation.
- ~30 minutes a week of extra practice for treated students overall; 45 minutes in year two among students actively in the sessions.
Gains were concentrated in year two, when practice was highest. They are in line with separate estimates of Khan Academy practice without Khanmigo, from a 2026 study published in PNAS. That benchmark is an outside comparison, not a third arm in this trial, and one of its co-authors is Oreopoulos himself. The authors' own summary is the line to keep: "Nothing in the achievement data requires the AI tutor to explain it."
Inside the treated group, the pattern points the same way. Each extra skill brought to proficiency per week was associated with 0.015 standard deviations higher achievement. Chats with Khanmigo showed a small association on their own, and it fell to roughly zero once skill mastery was accounted for. The authors flag this as descriptive, not causal, and it should be read that way.
The usage gap
The second-year message logs are the method worth copying, because they join two events: when a student was stuck, and whether the student asked the tutor. They cover 563 students across 16 schools and 100,017 exercise sessions, 56,362 of which included a mistake.
| Measure | Result |
|---|---|
| Ever messaged Khanmigo | 96% of students did |
| Practice days with a message (median student) | 33% |
| Exercise sessions with a message (median student) | 14% |
| Mistake sessions with a message (median student) - in the authors' words, "the moments a tutor is nominally for" | 17% |
Across all mistake sessions, 9,362 of 56,362 had any message, and the median number of messages in a mistake session was zero. This was the year of strongest implementation, so, the authors write, "if anything these figures overstate steady-state engagement."
What students sent matters as much as how often. Of messages sent during exercises, 39.4% were bare answers to Khanmigo's own questions, 24.2% were clicks on suggested help prompts, 12.6% were low-effort replies such as "idk" or requests for the answer, and 9.3% were off-task. Messages that contained a mathematical question or a step of reasoning made up 14.5% - about one in seven.
A rollout report built on the first row would have called this a success. The other three describe a tool that was rarely asked for help when help was the point. Which of those four numbers does your own AI rollout report?
What the vendor tried, and what moved
On 29 January 2026, between two delivery windows, Khan Academy changed Khanmigo to open by default shortly after each exercise page loaded. It is the obvious fix, and because it landed inside the trial, the logs caught it. The authors report "no detectable change around this switch" in the share of sessions with a message, the share of mistake sessions with a message, or messages per engaged session.
That is a before-and-after reading within one school year, not a randomized test of defaults. We found no other measured result on the question.
Khan Academy has since rolled out a broader redesign that activates the tutor throughout practice. A spokesperson told Chalkbeat the change was made because "students were not seeking out Khanmigo's help as much as we had hoped." Sal Khan, interviewed for the same Chalkbeat article (April 2026): "For a lot of students, it was a non-event ... They just didn't use it much." Khan Academy's chief learning officer, Kristen DiCerbo, named the mechanism in one line: "Students aren't great at asking questions well."
What did move was practice. Building the platform into mandatory, supervised session time produced about 30 minutes a week of it, in a setting where voluntary deployments, the authors note, generate almost none. A separate pair of randomized trials on an AI literacy platform for elementary students (Robinson and colleagues, June 2026) added an in-person tutor whose job was engagement, not teaching. Engagement rose 71 to 80%. Use stayed low, a few minutes a week, and reading scores did not improve.
What it cost
The marginal cost over the control classes was the license: about $15 per student per year, because the district already staffed the sessions. The authors credit the program with gains "at a cost per unit of learning far below human tutoring" - and on their own reading, those gains came from structured practice.
Their sentence on the AI component is the one an operator should pin up: the tutor, "included in the license but contributing no detectable achievement effect as deployed—represents capability districts have already paid for."
What transfers, and what does not
Remedial middle-school math is not a workplace. The 17% is not a forecast for employees with a coding assistant or a support copilot. We found no public, measured point-of-need usage rate for an enterprise AI assistant to compare it with, so the comparison stays a question for each organization's own data.
The mechanism is not specific to children. Help from a conversational tool has to be asked for. Asking requires knowing what to ask, and the people who most need help are the least likely to seek it unprompted - especially for small, incremental problems, where the cost of asking is immediate and the benefit is uncertain. The authors conclude that "the binding constraint appears to be engagement," and they are explicit that they cannot rule out some contribution from the tutor where it was used, or that richer designs - voice, awareness of the student's work, memory - would change the picture.
The two numbers to report instead
For any AI assistant already rolled out, the share of people who tried it is the least useful number available. Two others answer the question the budget depends on.
- Point-of-need use. Of the moments when someone was stuck - an error, a failed check, a ticket they escalated - in what share did they open the assistant? Where the stuck moments and the assistant's sessions are both logged, joining the two is the work.
- Outcome against the non-AI baseline. Does the result the business cares about beat the same workflow without the AI layer? If the gain matches the baseline, the AI is a cost line, not yet a contributor.
If the first number is low, this trial is a warning about the usual fixes. Protected, supervised time raised practice, but the 17% was measured under exactly that condition. One default switch inside the trial did not move it. The authors still hold that engagement "is responsive to design" - they list defaults, structure, facilitation and interface - and they leave open whether a more capable tutor would be asked more often. In this deployment, they write, "the constraint was not what the tutor could do but how often students engaged it."
So the second number decides the budget. If the outcome matches the non-AI baseline, the gain is most likely coming from the workflow, and the AI is capacity already paid for and not yet used. The test worth running before buying a bigger model is a design where the assistant acts in the stuck moment without waiting to be asked. That is the direction Khan Academy's own redesign has taken, and this trial did not get to measure it.
Sources
- Oreopoulos, P. and Low, N. (2026). One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment. EdWorkingPaper 26-1551. https://edworkingpapers.com/ai26-1551 (PDF: https://edworkingpapers.com/sites/default/files/ai26-1551.pdf; DOI: https://doi.org/10.26300/kner-hv33)
- Barnum, M. (2026, April 9). Why Sal Khan's AI revolution hasn't happened yet, according to Sal Khan. Chalkbeat. https://www.chalkbeat.org/2026/04/09/sal-khan-reflects-on-ai-in-schools-and-khanmigo/
- Robinson, C. D., Gormley, D., Ribeiro, A. T. and Loeb, S. (2026). Access is Not Enough: Human Support Improves Engagement with AI Tutoring. EdWorkingPaper 26-1451. https://edworkingpapers.com/ai26-1451
