Dartmouth Tests AI-Graded Homework: Students Call It Rigid — But the Ones Who Use It Score Higher
- Dartmouth College piloted an AI learning platform called Phosphor with 151 students in an intro stats course, as a fully voluntary, ungraded textbook substitute, with Claude Sonnet 4.6 grading the short-answer questions.
- 90.2% of students used the platform at least once — more than 6x the reading-completion rate students and instructors self-reported for the traditional textbook (10 to 15%).
- Comparing full engagement (all 24 lessons plus passing all 3 cross-lesson module reviews) against zero engagement, the final-exam gap was 0.71 standard deviations after controlling for prior performance, or 1.30 SD uncontrolled.
- Because of student feedback, three modules switched exam format twice, accidentally creating a natural experiment: only the module requiring students to write their own answers (short-answer) kept a "more practice, more points" relationship; the pure-multiple-choice module saw that relationship nearly vanish.
- The platform's built-in AI Q&A chat sidebar was queried only 72 times all semester, with just 14 students using it more than once.
Stuff AI Grading Into the Textbook — 90% of Students Opt In Anyway
A statistics course team at Dartmouth College recently piloted an AI learning platform called Phosphor with 151 students, weaving short-answer and multiple-choice quizzes directly into the reading flow, with short-answer responses graded by Claude Sonnet 4.6 against a rubric the instructor set.
What this platform is trying to solve is a ready-made contradiction. On one side, college students barely read their textbooks: completion rates have been sliding since the 1980s, students resist reading assignments, and objectively measured completion runs far below self-reports. In this course (MATH 010, Intro Statistics), students self-reported about 15% completion, the instructor estimated only 10%, and individual student comments ranged from "literally no one reads this" to "wait, this needs to be recorded too?"
On the other side, letting students use AI freely can actually hurt learning. In a randomized controlled trial with nearly a thousand students, Bastani et al. found that giving students unrestricted access to GPT-4 actually dropped scores by 17% once the tool was taken away — students had leaned on it as a crutch rather than a learning tool; only the version with instructional guardrails avoided this side effect. Meanwhile, the share of students using AI for coursework keeps climbing: a 2026 survey by the Higher Education Policy Institute (HEPI) found 94% of college students had used generative AI on graded assignments, up from 53% just two years earlier.
What Phosphor (formerly named Spongium) is trying to do is turn "actively answering" into a structural step of the reading experience itself: reading the text and taking an AI-graded mini-quiz become one motion. The design premise is that AI is most useful when embedded directly into the content-delivery system itself — which also echoes the well-documented "doer effect" in education research, covered below.
Read the Lesson, Quiz Right Away, AI Grades on the Spot
Phosphor is a web app. Course content is organized into "lessons," each backed by a bank of 15 to 20 questions. After a student finishes reading a lesson, they take a mini-quiz on the same page — machine and AI grade it instantly.
A CRQ (constructed-response question, i.e. a short-answer question) doesn't ask students to pick from options — it asks them to write out an answer in their own words, which the AI then compares against the instructor's rubric and reference answer to grade and explain. It requires one more step than multiple choice: organizing the answer in your own language.
The design is betting on the "doer effect": doing practice problems produces far more learning than passive reading alone. What actually drives the effect is the act of "actively answering" itself, not how many pages got read. Weaving practice into the reading flow is an attempt to make that effect the default. Whether that bet paid off is exactly what the natural experiment in the next section answers directly.
Short-Answer vs. Multiple Choice — Which One Actually Moves the Score
The three modules were supposed to use the same exam format, but student feedback forced two mid-course switches — and the result was something close to a controlled experiment.
Module 1 (Descriptive Statistics, 9 lessons) used a mixed quiz of short-answer plus multiple-choice. Module 2 (Probability & Sampling, 8 lessons), after a wave of student feedback that auto-graded short-answer felt "too rigid and demoralizing," cut quizzes down to pure multiple choice. By Module 3 (Statistical Inference, 7 lessons), the team had analyzed exam results, concluded that pure-multiple-choice quizzes were barely producing any learning gain, and added short-answer back.
So the same group of students, in the same course, experienced different exam formats across different modules. Plotting "lessons completed" against "exam score gained" gives three modules with completely different slopes.
The only difference across the three modules is exam format: when short-answer is present, the "more lessons completed → higher final score" relationship shows up; switch to pure multiple choice and it vanishes (R² drops from 0.123 to 0.001); switch back to short-answer and it returns. What's actually driving the score is the act of having to write out an answer in your own words. This lines up with testing-effect research: Kang et al. found that feedback-backed short-answer questions beat multiple choice on long-term retention (d=0.41).
See the full regression numbers for all three (Table 2)
| Sample | Module 1 → Midterm 1 (Short-Answer + MCQ) | Module 2 → Midterm 2 (Pure MCQ) | All 24 lessons → Final |
|---|---|---|---|
| All platform users | 1.64x + 77.9, R²=0.123 | 0.90x + 76.0, R²=0.027 | 0.41x + 84.7, R²=0.091 |
| ≥1 lesson completed | 1.37x + 80.0, R²=0.111 | −0.29x + 84.7, R²=0.001 | 0.45x + 84.0, R²=0.096 |
The apparent positive slope in Module 2's "all users" row is just an artifact of the gap between zero-completion and non-zero-completion students; restrict to students with completion records and the slope flips negative.
What Actually Moves the Score: Cross-Lesson Reviews
Lining up every comparison, the single biggest effect isn't the per-lesson quiz — it's the review test that spans a whole module. Students who passed all three module reviews scored 7.1 points higher on the final than those who didn't.
Cohen's d converts the difference between two groups' scores into "how many standard deviations apart," making it comparable across studies. In education research, d=0.66 is generally considered a medium-to-large effect, well above what most classroom interventions produce. The d=0.66 bar below is the single largest effect in the whole paper.
Of the five preregistered comparisons, only "passed all three reviews" held up after Holm correction for multiple comparisons (p<0.0001). This group is also the most self-selected in the whole study, so the authors anchor a firmer piece of evidence on a cleaner comparison: within Module 2, students who passed the review scored 6.1 points higher on the midterm than those who didn't (d=0.39), with content, timing, and class all held fixed — direction and magnitude both check out.
Module reviews differ from single-lesson quizzes in several ways: they span multiple lessons, interleave different topics, have a higher bar (90%), and offer an optional short-answer format. It's impossible to isolate which factor matters, but the most plausible driver is cross-topic interleaved retrieval practice, combined with the spaced-review rhythm students developed on their own.
Students Invented Their Own "Spaced Repetition"
Across the three module reviews, two-thirds of retake attempts (157 of 237) came back for at least one more retake — and most of those retakes happened after a delay, rather than immediately.
The median retake gap was about 1.5 days, and the retake rate was similar across the three modules (61%, 77%, and 56% respectively). This "come back a day later" pattern is exactly the spaced-review behavior memory research recommends — and students formed it entirely on their own, with no requirement to do so.
Full Participation vs. Zero Participation: How Big Is the Gap
Putting the two participation measures — lessons completed and reviews passed — into a single model estimates the final-exam gap between full and zero participation, using a Tobit model.
27% of students hit the ceiling with a perfect score on this final. A plain regression would treat those capped students as no different from other high scorers, understating the real gap. A Tobit model is built exactly for this "a batch of people hit the ceiling" situation, converting back the scores that "would have gone higher, but got capped at 100."
The result gives a range depending on whether you strip out students' prior ability:
Without controls, full participation (all 24 lessons plus passing all 3 reviews) beats zero participation by 14.7 points on the final (1.30 standard deviations). Once you control for both midterm scores, the gap is cut roughly in half, to 8.0 points (0.71 standard deviations).
The authors stress that 0.71 is a conservative lower bound, not the best estimate. Because the final covers the same content, the two midterms are essentially parallel measurements of the same underlying outcome — controlling for them also strips out learning gains Phosphor had already produced earlier in the semester. So the reasonable effect range runs from 0.71 SD (over-controlled) to 1.30 SD (contaminated by self-selection). By the standards of observational studies of educational interventions, that's not a small effect.
See the full three-column Tobit model (Table 4)
| Coefficient | Participation only | +Midterm 1 | +Midterm 2 |
|---|---|---|---|
| Lessons completed (0–24) | 0.074 (1.8) | 0.060 (1.8) | 0.058 (1.6) |
| Reviews passed (0–3) | 0.584 (1.8) | 0.162 (0.6) | 0.180 (0.7) |
| Midterm control | — | 15.55 (8.0) | 11.63 (6.9) |
| Full vs. zero participation gap (/100) | 14.7 | 8.0 | 8.0 |
| Converted to standard deviations | 1.30 | 0.71 | 0.71 |
After controlling for midterms, the review-passing coefficient gets largely absorbed (z≈0.6), while the per-lesson-completed coefficient barely moves (z≈1.8): the review's payoff has already been captured by the midterms, while lesson dosage still carries an independent end-of-semester signal.
Almost Nobody Opened the AI Q&A Sidebar
The platform also ships with an AI Q&A chat sidebar built on retrieval-augmented generation (RAG). Its usage lags the quiz feature by two orders of magnitude.
A RAG (retrieval-augmented generation) chat assistant converts a student's question into a vector, matches it against the course's content library, and feeds the most relevant passages into the AI's answer material — so the AI answers strictly based on this course's content, without going off-topic or making things up. It sits in the sidebar of the reading page, available anytime.
Over the whole semester, the chat sidebar was queried only 72 times total, and only 14 students used it more than once. Students gave two reasons: general-purpose AI tools felt faster and stronger for their questions, and the course content was already thorough enough that they hadn't built up enough questions while reading to bother opening a dedicated chat box.
This dovetails with outside observations: Khan Academy has previously reported that only 15% of users regularly engage with their tutoring chatbot. Embedding AI directly into the core "quiz plus feedback" assessment loop gets far more real student use than bolting it on as an external chat window.
How Much Should You Trust This — and What Are the Authors Doing Next
The authors are upfront about the limitations. This is an observational study, piloted in a single course at one elite institution, with no randomized control.
The biggest threat to interpretation is self-selection: students willing to do more practice may simply already be stronger and more motivated. The authors use the two midterm scores to control for prior ability, giving an upper and lower bound for the effect, but the cross-module comparison of lesson dosage is still tangled up with content domain, timing, and the simultaneous introduction of module reviews — and the "passed all three reviews" group is the most self-selected sample in the whole study. The survey data also comes from a small sample and may carry social-desirability bias.
A potential trade-off also surfaced: after short-answer was turned off, quiz completion rose from 49.6% to 58.8%; once it was turned back on, it dropped to 33.4%. The feature with the strongest effect isn't necessarily the one students are most willing to use.
The authors' next steps are to tie lesson completion to course grades for a cleaner "dose vs. effect" measurement, and to replicate the pilot across intro courses at more schools. A randomized comparison between short-answer and multiple-choice is also planned, which would help nail down the causal claim that short-answer is genuinely more effective.
These results suggest that high engagement and measurable learning gains can be achieved simultaneously; AI's ability to grade short-answer responses against a rubric at scale is pedagogically meaningful — not merely a convenience for instructors. Phosphor paper, abstract and discussion (iTextbooks 2026)
AI Grading Gets Woven Into the Textbook, 90% of Students Opt In — What Actually Moves the Score Is "Forcing You to Write the Answer Yourself"
Dartmouth's real-world test of the Phosphor AI learning platform with 151 students: short-answer questions move the needle more than multiple choice, and almost nobody clicked the AI Q&A sidebar. One page, illustrated, covers it all.
↓ Read this page in full · One chart moves
College students barely read their textbooks, and letting them use AI freely can hurt learning too. This study tries to build an AI learning tool that students actually opt into voluntarily — and that genuinely raises scores — to see if it can have it both ways.
✘ Yet nobody reads the textbook anyway (completion rate 10–15%), and unrestricted general AI use drops scores 17% once the tool is taken away
A randomized controlled trial with nearly 1,000 students found students treat AI as a crutch, not a study partner — and no matter how good a textbook is, it's worthless unopened. Dartmouth's stats course team asked: can you embed AI into the textbook itself, force out "active answering," and get students to use it voluntarily while actually learning?
The platform is called Phosphor: finish reading a lesson, immediately take a 4-question random mini-quiz — multiple choice auto-graded, short-answer (write out an answer in your own words, not pick an option) graded by Claude Sonnet 4.6 against the instructor's rubric, with reasons given. Fully voluntary, ungraded — and 90% of students used it anyway.
But "used a lot" doesn't necessarily mean "learned more" — couldn't this just be that students who already love studying also love doing extra practice? An accidental exam-format switch forced out the answer.
The three modules were supposed to use the same exam format, but student feedback forced two switches: Module 1 short-answer + MCQ → Module 2 switched to pure MCQ at students' request → Module 3 switched back to short-answer. The same group of students accidentally created a natural control.
XiaoHu here did the same thing across all three modules: more practice. Only in the module that required "writing your own answer" did more practice translate into a higher score; switch to picking options and that relationship disappears outright, then it comes back once short-answer returns. The small R² number in the chart scores "how tight this relationship is" — bigger means tighter, near 0 means no relationship at all.
The official numbers come in "standard deviations," which nobody has a feel for. Converted to class rank, it clicks instantly.
- × Nobody reads the textbook · completion only 10–15%
- × Free-for-all AI as a crutch · scores drop 17% once it's taken away
+1.64 pts on midterm
and 90% actually use it