AI tutors and learning outcomes Motivation
A carefully designed AI tutor beat an active-learning physics class on a short-term test
In short: In a Harvard introductory physics course, students learned more on a short post-test when a lesson was delivered by a custom-built AI tutor at home than when the same material was taught as an in-class active-learning lesson. The tutor was not a plain chatbot: instructors wrote question-specific prompts and solutions and structured each step.
| Paper | AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting |
| Authors | Greg Kestin, Kelly Miller, Anna Klales, Timothy Milbourne, Gregorio Ponti |
| Published | 3 June 2025 · Scientific Reports 15:17458 |
| Study type | Randomized controlled trial with a crossover design · Peer-reviewed |
| What we read | Full text read. We read the open-access full text. |
| Summarized |
What they did
Each student had one lesson (surface tension or fluid flow) with the AI tutor at home and the other as an in-class active-learning lesson, with the order swapped between two groups. Both versions followed the same pedagogy. The AI tutor used GPT-4 with instructor-written, question-specific prompts and provided solutions, so it did not rely on the model to work out answers itself. Students took a pre-test and a post-test for each lesson and answered survey questions on engagement and motivation.
Who took part
194 undergraduates in Physical Sciences 2, an introductory physics course for life-science students at Harvard University, autumn 2023.
What they found
- Median learning gains in the AI-tutor condition were more than double those in the active-learning class.
- Mean post-test score: 4.5 with the AI tutor versus 3.5 in class.
- Students in the AI condition spent a median of 49 minutes on the lesson, against a 60-minute class.
- Students reported feeling more engaged and more motivated with the AI tutor.
Effect size: Effect size about 0.63 standard deviations in the main regression; 0.73 to 1.3 SD in analyses designed to account for the ceiling on post-test scores.
Limitations
- One course at one highly selective university, two lessons.
- The post-test came straight after the lesson, so long-term retention was not measured.
- The AI tutor was carefully engineered by experienced instructors; results may not carry over to general-purpose chatbots.
- Engagement and motivation were self-reported.
What this means for learners and parents
- A well-structured AI tutor can teach a first pass of new material efficiently, at least for motivated university students.
- The design did a lot of the work: step-by-step scaffolding and prepared solutions, not open-ended chat.
- For lasting learning, it still makes sense to come back to the material later and test yourself without help.
Put it into practice (free tools)
Try it: Ask Poppy is an AI tutor for iPhone, iPad and Mac that asks questions and gives one hint at a time instead of handing over the answer. Free with 30 messages a day, then $3.99/week or $9.99/month. (We make it.)
Citation
Greg Kestin, Kelly Miller, Anna Klales, Timothy Milbourne, Gregorio Ponti (2025). AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports 15:17458. https://doi.org/10.1038/s41598-025-97652-6
Related summaries
- Unrestricted ChatGPT help raised practice scores but lowered exam scores · Hamsa Bastani et al., 2025 · Preregistered randomized controlled trial
- Supervised AI tutoring matched human tutors in UK maths classrooms · LearnLM Team (Google) et al., 2025 · Exploratory randomized controlled trial
- An AI coach for human tutors raised topic mastery, most for weaker tutors · Rose Wang et al., 2024 · Preregistered randomized controlled trial
- Six weeks of teacher-guided AI tutoring improved English results in Nigeria · Martin De Simone et al., 2025 · Randomized controlled trial