AI tutors and learning outcomes "AI brain rot" and cognitive offloading Socratic and dialogic tutoring
Unrestricted ChatGPT help raised practice scores but lowered exam scores
In short: In a high school in Turkey, students who practised maths with a plain ChatGPT-style GPT-4 assistant did much better during practice, then scored 17% lower than students without AI when they sat a test on their own. A second version with teacher-written prompts that gave hints instead of solutions avoided that drop, but did not beat the no-AI group on the test either.
| Paper | Generative AI without guardrails can harm learning: Evidence from high school mathematics |
| Authors | Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, Rei Mariman |
| Published | 25 June 2025 · Proceedings of the National Academy of Sciences (PNAS) 122(26) |
| Study type | Preregistered randomized controlled trial (field experiment) · Peer-reviewed |
| What we read | Full text read. We read the open-access full text (PubMed Central PMC12232635). |
| Summarized |
What they did
Researchers ran four 90-minute maths review sessions in about fifty classes. Each session had an assisted practice phase and then an unassisted exam. Classes were randomly assigned to one of three arms: textbook only (with solutions shown after answering), "GPT Base" (a standard GPT-4 chat interface), or "GPT Tutor" (GPT-4 with problem-specific teacher input, including the solution and suggested hints, and instructions to guide rather than give answers). The main analysis was preregistered.
Who took part
Nearly 1,000 students in grades 9, 10 and 11 at one large high school in Turkey, during the autumn semester of the 2023–24 school year. The sessions covered about 15% of each grade's maths curriculum for the semester.
What they found
- During assisted practice, GPT Base raised scores by 48% and GPT Tutor by 127% compared with the textbook-only group.
- On the unassisted exam, the GPT Base group scored 17% lower than the textbook-only group, a statistically significant drop.
- The GPT Tutor group's exam scores were not significantly different from the textbook-only group: the harm was largely avoided, but there was no measurable gain.
- Chat logs showed GPT Base students often asked for and copied solutions. GPT Base also often gave wrong answers. Students did not perceive that copying had hurt their learning.
Effect size: Practice: +48% (GPT Base) and +127% (GPT Tutor). Unassisted exam: −17% (GPT Base); GPT Tutor about the same as control. Intention-to-treat estimates relative to the textbook-only arm.
Limitations
- One school in one country, and maths only.
- Short exposure: four sessions.
- GPT Tutor depended on labour-intensive, problem-by-problem teacher prompts; a general chatbot with a "study" setting is not the same thing.
- The exam measured short-term learning within the same sessions, not long-term retention.
- The published article carries a correction notice; the figures here are from the corrected open-access version.
What this means for learners and parents
- Practice that feels successful with AI is not the same as learning. Check what your child, or you, can do without the AI.
- How the AI is set up matters: an assistant that hands over solutions invites copying, while one built to hint and check reduces the harm.
- A sensible home rule: try first, ask for one hint at a time, then do a similar problem with no AI.
Put it into practice (free tools)
- Check whether a prompt hands over the thinking
- Is using AI for homework cheating? A parent's guide
- AI tutors that guide instead of answering
Try it: Ask Poppy is an AI tutor for iPhone, iPad and Mac that asks questions and gives one hint at a time instead of handing over the answer. Free with 30 messages a day, then $3.99/week or $9.99/month. (We make it.)
Citation
Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, Rei Mariman (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences (PNAS) 122(26). https://doi.org/10.1073/pnas.2422633122
Related summaries
- A carefully designed AI tutor beat an active-learning physics class on a short-term test · Greg Kestin et al., 2025 · Randomized controlled trial with a crossover design
- Supervised AI tutoring matched human tutors in UK maths classrooms · LearnLM Team (Google) et al., 2025 · Exploratory randomized controlled trial
- An AI coach for human tutors raised topic mastery, most for weaker tutors · Rose Wang et al., 2024 · Preregistered randomized controlled trial
- Six weeks of teacher-guided AI tutoring improved English results in Nigeria · Martin De Simone et al., 2025 · Randomized controlled trial