Manaz Afzal

Education · Colleges and training providers

Students at risk of leaving, flagged while there is still time to help

A termly list for student-support advisers: who to contact first, and why. Tested on 4,424 undergraduate records. After the first semester, the top fifth of the list contains 57% of the students who later dropped out, and 92% of the students on it did leave.

57%of eventual dropouts in the top 20% after semester one
48%already at enrolment, before any grades exist
39%for the usual rule, "fees overdue or in debt", which flags 18% of students

Executive summary

After the first semester, the top fifth of a termly list holds 57% of the students who later dropped out, and 92% of the students on it did leave.

57%of eventual dropouts in the top 20% after semester one
48%already at enrolment, before any grades exist
39%for the usual fees-overdue rule, which flags 18% of students

Situation

Two or three advisers support a few thousand students and hear about trouble when fees go overdue or a census date passes, which is often too late.

Approach

Two scores from one pipeline, at enrolment and after semester one, on 4,424 undergraduate records, scored out-of-fold; gender, nationality and marital status excluded.

Result

The semester-one score reaches 57% of dropouts in a fifth of the intake against 39% for the fee rule. The enrolment score is weaker and arrives a semester earlier.

The business problem

Student-support advisers at a university or TAFE can contact only part of each intake. Early-alert programmes such as Georgia State’s, which prompt an adviser when a student’s record shows a risk, are credited with higher retention and graduation, but they depend on alerts that are few enough to act on and arrive before the student has gone.

I measured how many eventual dropouts a termly adviser list catches at enrolment and again after first-semester results, at a list size an advising team can work, against the fees-overdue rule many institutions use.

Team
2 to 3 student-support advisers
Scale
An intake of 500 to 5,000 students a year
Systems
Student record system such as Callista, TechnologyOne Student or Banner, a CRM for advising such as Salesforce Education Cloud
Constraint
Privacy Act 1988 and state education rules; flags inform an adviser and never change grades, fees or enrolment

What was built

Two scores from the same pipeline. The first runs at enrolment from application details, prior qualifications, admission grade, attendance mode, age and fee status. The second runs after first-semester results, adding units enrolled, assessed and passed, and grades. Each produces a ranked list with plain reasons ("passed 2 of 6 units; fees overdue; evening attendance") in the adviser's queue in the student management system or a shared sheet.

Student systemEnrolment, fees, units,results each censusScoreAt enrolment and aftersemester-one resultsAdviser listRanked, with reasons;sized to the teamContactCall, meeting orstudy-support referralRecordOutcome noted; nextterm's refit uses it
Termly workflow. Gender, nationality, international status and marital status are excluded from both scores.
0%25%50%75%100%0%20%40%60%80%100%After semester oneAt enrolmentRandomAfter semester one, top 20%:57% of dropouts, 92% precisionFees-overdue rule: 39%Share of the cohort flagged for an adviser conversation
Out-of-fold scores: every student was scored by a model that never saw them. The dot marks the fees-overdue rule, which flags a fixed share of students. Data: UCI 697, Polytechnic Institute of Portalegre, 4,424 students

Results

ListStudents flaggedDropouts caughtFlagged who left
Enrolment score, top 10%44229.2%93.9%
Enrolment score, top 20%88448.5%77.9%
Semester-one score, top 10%44230.7%98.6%
Semester-one score, top 20%88457.5%92.4%
Fees overdue or debtor78539.0%70.6%
Random 20%88520.0%32.1%

AUC is 0.822 at enrolment and 0.903 after semester one, against 0.656 for the fee rule. Including the excluded personal attributes would raise the semester-one AUC only to 0.906.

At enrolmentAfter semester oneTop 10% flagged31%Top 20% flagged57%Top 30% flagged74%
Waiting for first-semester results adds about nine points at the top-20% mark. The enrolment score is weaker but arrives a semester earlier, when support is cheapest.

The signal in first-semester results

0 units passed718 students79%1 units passed127 students73%2 units passed160 students70%3 units passed269 students52%4 units passed433 students36%5 units passed723 students20%6 units passed1,171 students10%7 units passed471 students7%8+ units passed352 students15%All students 32%
Students who pass no first-semester units drop out at 79%. Dropout falls steadily with each unit passed until five or six.
Fees not up to date528 students87%0 units passed in semester 1718 students79%Debtor503 students62%Aged 25+ at enrolment1,138 students55%Evening attendance483 students43%Scholarship holder1,099 students12%All units passed in semester 11,728 students9%All students 32%
Fee status is the strongest single signal at enrolment: 87% of students behind on fees dropped out. Scholarship holders drop out at 12%.

What it means for a provider

For an intake of 1,000 with this dropout rate, about 321 students will leave. The semester-one list of 200 contains about 185 of them. If adviser contact keeps one in five of those enrolled for another year, that is about 37 students; at an annual fee of $15,000 that is roughly $550,000 in tuition.

The retention effect of the contact itself is not measured in this data. A provider should test it: contact half of the flagged list in the first term and compare re-enrolment with the other half.

Data and method

Source
UCI Machine Learning Repository 697, "Predict Students' Dropout and Academic Success" (Realinho et al., 2022)
Size
4,424 students: 2,209 graduated, 794 still enrolled, 1,421 dropped out
Setting
Polytechnic Institute of Portalegre, Portugal; 17 undergraduate degrees; intakes 2008/09 to 2018/19
Validation
5-fold stratified cross-validation, 3 seeds, out-of-fold scores

Gradient-boosted classifiers predict dropout against graduating or still being enrolled. The enrolment score uses 20 attributes; the semester-one score adds six first-semester attributes. Second-semester data is held back entirely so that the second score is available mid-year.

Limitations

What a provider would do next

Technical depth

The two checkpoints share preprocessing so their scores are comparable. Excluded attributes were tested both ways and reported, so the cost of leaving them out is visible. Precision at a fixed list size is reported because adviser time, not a probability threshold, sets the list length.

Analysis code (Python, scikit-learn)

Financial impact

About $550,000 in tuition kept for an intake of 1,000

Value is students kept enrolled for another year because an adviser reached them in time.

InputValueSource
Intake1,000
Students who will leave321Measured rate measured in this data
Leavers on the semester-one list of 200185Measured
Share kept by adviser contact20% not measured here
Annual fee$15,000

Calculation

  1. Leavers reached185
  2. Students kept a yearabout 37
  3. Tuition keptabout $550,000

Tuition kept a yearabout $550,000

The effect of the contact itself is not in this data; contact half the list in the first term and compare re-enrolment.

Implementation and rollout

One term from student records to an adviser queue in week three

  1. Weeks 1 to 2

    Extract

    Three to five years of enrolment, results and withdrawals from Paradigm, TechnologyOne or Callista.

    Student services, Manaz

  2. Weeks 3 to 4

    Refit and test

    Train on earlier intakes and test on the latest, against the fee rule.

    Manaz

  3. Term 1

    Run with a split

    Half the flagged list contacted; re-enrolment compared with the other half.

    Advisers

  4. Then

    Decide

    Keep, resize or stop on re-enrolment gained against adviser time.

    Head of student services, Manaz

Tools

Student management system exports, Moodle or Canvas activity, if available, Python gradient boosting, termly, Adviser queue in the SMS or a shared sheet

Team

  • Head of student services owns the policy and list size
  • Advisers contact students and record outcomes
  • Manaz pipeline, scores, reasons and termly review

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Flag treated as a judgement on the studentFlags inform an adviser only and never change grades, fees or enrolment.
Bias through personal attributesGender, nationality and marital status excluded; including them adds 0.003 AUC.
Model from one Portuguese institutionRefitted on the provider’s own records before any use.
Quiet leavers who pass and do not re-enrolRe-enrolment check before census as a second trigger.

Key takeaways

Key takeaways

  • Students who pass no first-semester units drop out at 79%.
  • Fee status is the strongest signal at enrolment.
  • Waiting for semester-one results adds about nine points at the top-20% mark.
  • Excluding personal attributes costs almost nothing in accuracy.