Education & Training
Students stuck on a skill, flagged to the teacher after three questions
Stuck-student alerts for a maths practice platform, tested on 7,288 real skill assignments against the two-wrong rule.
Executive summary
A score after three questions flags a fifth of students and reaches 35% of those who get stuck, with 77% of flags correct against 58% for the two-wrong rule.
Situation
Teachers learn a student is stuck on a skill from the end-of-week report.
Approach
A stuck score after the third question, tested on 7,288 real ASSISTments skill assignments.
Result
35% of stuck students reached by checking a fifth, against 27% for the rule at the same count.
The problem
On a maths practice platform a teacher sees a class report after the lesson, and students who are stuck on a skill keep answering until they give up. Platforms such as Mathspace and ASSISTments give teachers class and mastery reports, and controlled trials of online maths homework show the biggest gains for lower-performing students when teachers act on what the reports show.
I wanted an alert a teacher could act on during the lesson: after a student’s third question on a skill, flag the ones unlikely to reach mastery, in order, and measure it against the common two-wrong-in-a-row rule.
- Team
- Head of maths, classroom teachers, learning support, the platform’s product team
- Scale
- Thousands of students and tens of thousands of skill assignments a year
- Systems
- ASSISTments, Mathspace, Education Perfect or another practice platform, the school LMS
- Measure
- Stuck students reached per teacher check; skills mastered after an intervention
What I built
A stuck score computed after each student’s third question on a skill. It reads whether each answer was right, the attempts, hints and time taken, how hard the skill has been for earlier students, and how the student fared on earlier skills. The teacher’s dashboard shows the flagged students for the lesson in order, with the reason beside each name.
Results
I tested on the ASSISTments 2009 to 2010 skill-builder data: real practice records for 4,217 students across 66,238 skill assignments. After three questions without mastery, 44% of students went on to miss mastery within ten questions. The score was fitted on the first 70% of the school year and tested on the 7,288 skill assignments that followed.
| Flag | Students flagged | Stuck students reached | Flags correct |
|---|---|---|---|
| Two or more of the first three wrong | 45% | 59% | 58% |
| Stuck score, same number | 45% | 64% | 63% |
| Stuck score, top fifth | 20% | 35% | 77% |
| Rule cut to a fifth at random | 20% | 27% | 58% |
The score separates stuck students from the rest with an AUC of 0.74, against 0.66 for counting wrong answers. Students who get stuck spend 7.7 questions on the skill on average before they stop.
Data and method
- Data
- ASSISTments 2009 to 2010 skill-builder data, corrected and collapsed release: first-attempt correctness, attempts, hints and response time for main questions, for 4,217 anonymised students in US middle schools.
- Target
- Student does not answer three in a row correctly within ten questions on the skill, including students who stop before mastery
- Decision point
- After the third question, for students not yet at mastery who reached a fourth question
- Methods
- Gradient-boosted classifier on the first three answers, attempts, hints, timing, the skill’s earlier stuck rate and the student’s earlier skills; fitted on the first 70% of the year by start time and tested on the rest
- Comparison
- Two or more of the first three answers wrong
Limitations
- US middle-school maths from 2009 and 2010 on one platform. The score is refitted on the platform’s own practice logs and curriculum.
- Some students stop for reasons outside the skill, such as the end of a lesson, and count as stuck here.
- The test measures who got stuck, not whether a teacher’s check helped; a term with half the classes using the flags measures that.
- The score reads practice behaviour only and does not rank students by background.
How I would take it live
- Export a year of practice logs from the platform with skill tags
- Fit the score per year level and agree a flag count per lesson with teachers
- Run a term with half the classes using the flags
- Adopt with mastery rates after a flag compared across the two groups