Manaz Afzal

Education & Training

Students stuck on a skill, flagged to the teacher after three questions

Stuck-student alerts for a maths practice platform, tested on 7,288 real skill assignments against the two-wrong rule.

77%of flags correct in the top fifth
58%for the two-wrong rule
35%of stuck students reached
0.74AUC after three questions

Executive summary

A score after three questions flags a fifth of students and reaches 35% of those who get stuck, with 77% of flags correct against 58% for the two-wrong rule.

77%of flags correct
35%of stuck students reached
0.74AUC after three questions

Situation

Teachers learn a student is stuck on a skill from the end-of-week report.

Approach

A stuck score after the third question, tested on 7,288 real ASSISTments skill assignments.

Result

35% of stuck students reached by checking a fifth, against 27% for the rule at the same count.

The problem

On a maths practice platform a teacher sees a class report after the lesson, and students who are stuck on a skill keep answering until they give up. Platforms such as Mathspace and ASSISTments give teachers class and mastery reports, and controlled trials of online maths homework show the biggest gains for lower-performing students when teachers act on what the reports show.

I wanted an alert a teacher could act on during the lesson: after a student’s third question on a skill, flag the ones unlikely to reach mastery, in order, and measure it against the common two-wrong-in-a-row rule.

Team
Head of maths, classroom teachers, learning support, the platform’s product team
Scale
Thousands of students and tens of thousands of skill assignments a year
Systems
ASSISTments, Mathspace, Education Perfect or another practice platform, the school LMS
Measure
Stuck students reached per teacher check; skills mastered after an intervention

What I built

A stuck score computed after each student’s third question on a skill. It reads whether each answer was right, the attempts, hints and time taken, how hard the skill has been for earlier students, and how the student fared on earlier skills. The teacher’s dashboard shows the flagged students for the lesson in order, with the reason beside each name.

Three questionsAnswers, attempts,hints and timingScoreChance of not masteringwithin ten questionsTeacher viewFlagged studentsin orderInterventionWorked exampleor a conversation
Flags reach the teacher during the lesson, while the student is still on the skill.

Results

I tested on the ASSISTments 2009 to 2010 skill-builder data: real practice records for 4,217 students across 66,238 skill assignments. After three questions without mastery, 44% of students went on to miss mastery within ten questions. The score was fitted on the first 70% of the school year and tested on the 7,288 skill assignments that followed.

0%20%40%60%0%10%20%30%Stuck scoreRandom listA fifth flagged:35% of stuck students,77% of flags correctShare of students on a skill flagged to the teacher
Flagging the top fifth reaches 35% of the students who get stuck, and 77% of those flagged are stuck. The top tenth is 87% correct. Data: ASSISTments 2009 to 2010 skill-builder data, Worcester Polytechnic Institute
Stuck score, top fifthreaches 35% of stuck students77%Two of three wrong, same numberreaches 27% of stuck students58%
At the same number of flags the rule reaches fewer stuck students and wastes more teacher checks.
FlagStudents flaggedStuck students reachedFlags correct
Two or more of the first three wrong45%59%58%
Stuck score, same number45%64%63%
Stuck score, top fifth20%35%77%
Rule cut to a fifth at random20%27%58%

The score separates stuck students from the rest with an AUC of 0.74, against 0.66 for counting wrong answers. Students who get stuck spend 7.7 questions on the skill on average before they stop.

Data and method

Data
ASSISTments 2009 to 2010 skill-builder data, corrected and collapsed release: first-attempt correctness, attempts, hints and response time for main questions, for 4,217 anonymised students in US middle schools.
Target
Student does not answer three in a row correctly within ten questions on the skill, including students who stop before mastery
Decision point
After the third question, for students not yet at mastery who reached a fourth question
Methods
Gradient-boosted classifier on the first three answers, attempts, hints, timing, the skill’s earlier stuck rate and the student’s earlier skills; fitted on the first 70% of the year by start time and tested on the rest
Comparison
Two or more of the first three answers wrong

Limitations

  • US middle-school maths from 2009 and 2010 on one platform. The score is refitted on the platform’s own practice logs and curriculum.
  • Some students stop for reasons outside the skill, such as the end of a lesson, and count as stuck here.
  • The test measures who got stuck, not whether a teacher’s check helped; a term with half the classes using the flags measures that.
  • The score reads practice behaviour only and does not rank students by background.

How I would take it live

  • Export a year of practice logs from the platform with skill tags
  • Fit the score per year level and agree a flag count per lesson with teachers
  • Run a term with half the classes using the flags
  • Adopt with mastery rates after a flag compared across the two groups

Financial impact

Teacher checks that reach a stuck student

The gain is teacher time spent on students who need it. Student and skill counts are assumptions; precision is measured.

InputValueSource
Students on the platform2,000
Skills practised a year each40
Flags correct, score77%Measured
Flags correct, rule58%Measured

Calculation

  1. Teacher checks a year7,054
  2. Checks reaching a stuck student, score5,446
  3. With the rule4,124

1,322 more checks a year reaching a stuck student

A term with half the classes using the flags measures the effect on mastery.

Implementation and rollout

One term from practice logs to tested flags

  1. Weeks 1 to 2

    Export

    A year of practice logs with skill tags.

    Platform team, Manaz

  2. Week 3

    Fit and set

    Score fitted; flag counts agreed with teachers.

    Head of maths, Manaz

  3. One term

    Compare

    Half the classes use the flags.

    Teachers

  4. After the term

    Adopt

    Mastery after a flag compared.

    Head of maths

Tools

ASSISTments, Mathspace or Education Perfect, School LMS, Teacher dashboard, Python, LightGBM

Team

  • Head of maths owns the rollout
  • Teachers act on flags
  • Platform team data and dashboard
  • Manaz score and review

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Labelling studentsFlags go to teachers only.
Curriculum changeScore refitted per year level.
Lesson-end stopsFlag shown during the lesson.
BiasPractice behaviour only.

Key takeaways

Key takeaways

  • Stuck students show within three questions.
  • Counting wrong answers wastes checks.
  • A fifth flagged holds 35% of stuck students.
  • Compare classes for a term before adopting.