Manaz Afzal
Education & Training

The students about to miss an assignment, on the tutor’s list a week before the deadline

Assignment outreach for an online university, tested on 68,333 real student-assignment records against the quiet-student trigger tutors use today.

45%of missed or failed work in the top fifth of the cohort
87%of students contacted at risk
34%for the quiet-student trigger at the same workload
0.82AUC one week before the deadline

Executive summary

An outreach score run a week before each deadline puts 45% of missed or failed assignments in the top fifth of the cohort, against 34% for the quiet-student trigger at the same workload.

45%of missed or failed work in the top fifth
87%of students contacted at risk
0.82AUC a week before the deadline

Situation

Tutors contact students who have gone quiet, with no order inside a list that covers a third of the cohort.

Approach

A score from learning-platform activity and assignment history a week before each deadline, fitted on 2013 and tested on 68,333 student-assignments in 2014.

Result

45% of missed or failed work reached by contacting a fifth of students, against 34% for the trigger.

The problem

At an online university, tutors have time for a limited number of calls and emails each week, and the usual trigger is a student who has gone quiet or has already missed an assignment. The Open University’s work with predictive learning analytics found that weekly at-risk flags help when tutors use them regularly and act on them, and do little when they do not, so the list has to be short and timely.

I wanted to measure how much missed and failed work a tutor reaches by contacting students a week before each deadline, ranked by learning-platform activity and assignment history, against the quiet-student trigger at the same workload.

Team
Head of student success, module leads, tutors and student support
Scale
Thousands of students per module presentation, several assignments each
Systems
Moodle, Canvas or Blackboard activity logs, the student record system, CRM for outreach
Measure
Missed or failed assignments reached per contact; submission rate after outreach

What I built

An outreach score run seven days before every tutor-marked assignment deadline. It reads each active student’s clicks in the learning platform over the last one, two and four weeks, days since their last activity and their record on the previous assignment, then ranks the cohort. Each tutor receives a list sized to the calls they can make that week, with the reason beside each name.

Learning platformDaily activityby studentScoreSeven days beforeeach deadlineTutor listSized to the week’soutreach capacityOutcomeSubmission recordedand fed back
Outcomes after each deadline become training data for the next one.

Results

I tested on the Open University Learning Analytics Dataset: real activity and assessment records for 25,152 students across 7 modules. The score was fitted on the 2013 presentations and tested on the 68,333 student-assignment pairs in 2014, where 39% of active students went on to miss or fail the assignment.

0%20%40%60%80%0%10%20%30%40%Outreach scoreRandom listA fifth of the cohort:45% of missed or failed workShare of active students on the tutor’s outreach list
Contacting the top fifth of the cohort reaches 45% of the missed or failed work, and 87% of the students contacted would otherwise have missed or failed. The score separates the two groups with an AUC of 0.82. Data: Open University Learning Analytics Dataset, Kuzilek, Hlosta and Zdrahal, Scientific Data 2017
Outreach listShare of cohortMissed or failed work reachedPrecision
No activity in the last seven days32%55%66%
Score, same list size32%61%74%
Score, top fifth20%45%87%
Quiet rule cut to a fifth at random20%34%66%

At the same workload the quiet-student trigger reaches 34%, because it flags 32% of the cohort with no order inside the list. When tutors can contact everyone the trigger flags, the score still reaches 61% against 55%.

The first assignment

Before the first assignment there is no history, and the score relies on activity alone: its AUC falls to 0.68. Risk then rises with each assignment, so the lists grow later in the module.

Assignment 115,812 student-assignments30%Assignment 214,984 student-assignments34%Assignment 314,245 student-assignments43%Assignment 412,267 student-assignments44%Assignment 57,874 student-assignments44%Assignment 63,151 student-assignments50%
Risk climbs from 30% at the first assignment to 50% at the sixth, which is where outreach capacity matters most.

Data and method

Data
Open University Learning Analytics Dataset (UCI Machine Learning Repository): daily learning-platform clicks, registrations and assessment submissions for 7 modules over 2013 and 2014, anonymised by the Open University. No student is identifiable.
Target
An active student who does not submit the tutor-marked assignment by its deadline, or scores below the pass mark of 40
Decision point
Seven days before each deadline, using only activity and results recorded up to that day
Methods
Gradient-boosted classifier on recent activity, days since last activity, assignment number, previous assignment outcome and module; fitted on 2013, tested on 2014
Comparison
No learning-platform activity in the last seven days, the usual tutor trigger

Limitations

  • One university’s distance-learning modules from 2013 and 2014. Activity patterns differ by platform and course design, so the score is refitted on the provider’s own logs.
  • The test measures who would have missed the assignment, not the effect of the call. A pilot with a comparison group measures how many outreach contacts turn into submissions.
  • The score uses activity and results only, not demographics, so it does not rank students by background.
  • A score is a prompt for a tutor, never a decision about a student.

How I would take it live

  • Export two years of activity logs and assessment records from the learning platform and student record system
  • Fit the score per faculty and agree list sizes with tutors
  • Run a pilot term with half the tutor groups using the list and half working as usual
  • Adopt with submission rates reviewed after every deadline

Financial impact

Tutor contacts that land on students at risk

The gain is outreach that reaches at-risk students. Cohort size and assignments per module are assumptions; capture and precision are measured.

InputValueSource
Students in a cohort4,000
Assignments per module5
Share contacted each deadline20%
At-risk share of those contacted87%Measured

Calculation

  1. Contacts a presentation4,000
  2. Contacts that reach an at-risk student3,477
  3. With the trigger at the same workload2,630

847 more at-risk students reached each presentation

Retention and fee income effects depend on how many contacts turn into submissions, which a pilot measures.

Implementation and rollout

One term from logs to a tested outreach list

  1. Weeks 1 to 2

    Export

    Two years of activity and assessment records.

    Student success, Manaz

  2. Week 3

    Fit and size

    Score fitted per faculty; list sizes agreed with tutors.

    Module leads, Manaz

  3. Pilot term

    Comparison

    Half the tutor groups use the list.

    Tutors

  4. After the term

    Adopt

    Submission rates compared and the list adopted.

    Head of student success

Tools

Moodle, Canvas or Blackboard logs, Student record system, CRM for outreach, Python, scikit-learn

Team

  • Head of student success owns outreach
  • Tutors make the contacts
  • Module leads list sizes
  • Manaz score and review

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Students labelledScore is a prompt for a tutor, never a decision.
Course design changesScore refitted each presentation.
Demographic biasActivity and results only.
Outreach fatigueContacts capped per student per term.

Key takeaways

Key takeaways

  • Missed assignments are visible a week ahead in platform activity.
  • A fifth of the cohort holds 45% of missed or failed work.
  • Ranking beats a yes-or-no trigger at the same workload.
  • A pilot term measures the effect of the call.