The problem
At an online university, tutors have time for a limited number of calls and emails each week, and the usual trigger is a student who has gone quiet or has already missed an assignment. The Open University’s work with predictive learning analytics found that weekly at-risk flags help when tutors use them regularly and act on them, and do little when they do not, so the list has to be short and timely.
I wanted to measure how much missed and failed work a tutor reaches by contacting students a week before each deadline, ranked by learning-platform activity and assignment history, against the quiet-student trigger at the same workload.
- Team
- Head of student success, module leads, tutors and student support
- Scale
- Thousands of students per module presentation, several assignments each
- Systems
- Moodle, Canvas or Blackboard activity logs, the student record system, CRM for outreach
- Measure
- Missed or failed assignments reached per contact; submission rate after outreach
What I built
An outreach score run seven days before every tutor-marked assignment deadline. It reads each active student’s clicks in the learning platform over the last one, two and four weeks, days since their last activity and their record on the previous assignment, then ranks the cohort. Each tutor receives a list sized to the calls they can make that week, with the reason beside each name.
Results
I tested on the Open University Learning Analytics Dataset: real activity and assessment records for 25,152 students across 7 modules. The score was fitted on the 2013 presentations and tested on the 68,333 student-assignment pairs in 2014, where 39% of active students went on to miss or fail the assignment.
| Outreach list | Share of cohort | Missed or failed work reached | Precision |
|---|---|---|---|
| No activity in the last seven days | 32% | 55% | 66% |
| Score, same list size | 32% | 61% | 74% |
| Score, top fifth | 20% | 45% | 87% |
| Quiet rule cut to a fifth at random | 20% | 34% | 66% |
At the same workload the quiet-student trigger reaches 34%, because it flags 32% of the cohort with no order inside the list. When tutors can contact everyone the trigger flags, the score still reaches 61% against 55%.
The first assignment
Before the first assignment there is no history, and the score relies on activity alone: its AUC falls to 0.68. Risk then rises with each assignment, so the lists grow later in the module.
Data and method
- Data
- Open University Learning Analytics Dataset (UCI Machine Learning Repository): daily learning-platform clicks, registrations and assessment submissions for 7 modules over 2013 and 2014, anonymised by the Open University. No student is identifiable.
- Target
- An active student who does not submit the tutor-marked assignment by its deadline, or scores below the pass mark of 40
- Decision point
- Seven days before each deadline, using only activity and results recorded up to that day
- Methods
- Gradient-boosted classifier on recent activity, days since last activity, assignment number, previous assignment outcome and module; fitted on 2013, tested on 2014
- Comparison
- No learning-platform activity in the last seven days, the usual tutor trigger
Limitations
- One university’s distance-learning modules from 2013 and 2014. Activity patterns differ by platform and course design, so the score is refitted on the provider’s own logs.
- The test measures who would have missed the assignment, not the effect of the call. A pilot with a comparison group measures how many outreach contacts turn into submissions.
- The score uses activity and results only, not demographics, so it does not rank students by background.
- A score is a prompt for a tutor, never a decision about a student.
How I would take it live
- Export two years of activity logs and assessment records from the learning platform and student record system
- Fit the score per faculty and agree list sizes with tutors
- Run a pilot term with half the tutor groups using the list and half working as usual
- Adopt with submission rates reviewed after every deadline