Manaz Afzal

Retail and e-commerce · Win-back

Win-back emails sent to the customers most likely to return

Monthly customer segments and a ranked win-back list for an online store, tested on a later period of 1,067,371 real order lines.

284returning buyers from 500 emails, against 259 by recency and 145 at random
77%of revenue from the top 20% of customers
0.789AUC on the later test window (recency 0.762)

Executive summary

Win-back emails chosen by a model reach 284 returning buyers from 500 sends, against 259 by recency and about 145 at random, on two years of real store orders.

284returning buyers from 500 win-back emails; recency reaches 259
0.789AUC on a later test window, against 0.762 for recency (gain 0.027, 95% CI 0.018 to 0.035)
77%of revenue from the top 20% of customers

Situation

An online store sends the same newsletter to everyone and its win-back flow to whoever bought last. Discounts leak margin to customers who would reorder anyway.

Approach

Four plain-named segments and a 90-day reorder score from order history, chosen on an earlier window and tested once on customers as at September 2011.

Result

The score beats recency modestly and reliably. Gradient boosting lost to logistic regression and was not used. Segments set the message; the score sets who gets it.

The business problem

An online store’s win-back emails usually go to everyone who has not bought in 90 or 180 days. Retailers using flows in Klaviyo or similar report that win-back works best when the timing and offer follow each customer’s own buying gap and order value, rather than one date for all.

I measured how many returning buyers a ranked monthly win-back list reaches per 500 emails against a recency rule and against sending at random.

Team
Owner plus one marketing or e-commerce coordinator
Scale
A few thousand customers, one to four campaigns a month
Systems
Shopify or WooCommerce orders, Klaviyo or Mailchimp, Xero
Constraint
Spam Act 2003 consent and unsubscribe rules; Privacy Act for customer data

What was built

A monthly job that reads order history, places every customer in one of four plain-named segments, and scores each lapsed customer's chance of buying in the next 90 days. Segments and scores sync to the email tool as tags; the top of the lapsed list gets the win-back flow. A second score reads on-site behaviour (pages viewed, product-page time, exits) to flag sessions likely to convert, for remarketing.

Shopify ordersOrder lines, customer,date, valueSegmentsRecency, frequency,spend: four groupsScore90-day return chanceper lapsed customerKlaviyo tagsSegment and scoreon each profileSendMarketer writes andapproves the offer
Monthly workflow. The model picks who; a person picks the message and the send.

Results

£0k£500k£1,000k£1,500k2009-122010-062010-122011-06RevenueNovember peak
Two years of trading, with the November peak that every gift seller plans around. Data: UCI Online Retail II, 1,067,371 invoice lines
0%25%50%75%100%0%20%40%60%80%100%Even splitCumulative revenueTop 20%: 77% of revenueShare of identified customers
Revenue is concentrated: the top 20% of customers bring 77% of identified revenue and the top 1% bring 32%.

Four segments, four actions

Share of customersShare of revenueLoyal big spenders67%Drifting24%Lapsed one-offs5%Regulars3%
Loyal big spenders are 17% of customers and 67% of revenue. Lapsed one-offs are 41% of customers and 5% of revenue.
SegmentCustomersShare of revenueBought again in 90 daysAction
Loyal big spenders87167.3%84.2%Account-style contact; early access, no discounts
Drifting1,60723.9%47.0%Win-back email with a reason to return
Lapsed one-offs2,1315.3%20.4%Low-frequency newsletter only
Regulars6473.5%56.3%Replenishment reminders

Who to email first

Random 500lapsed customers145Most recent firstcommon email-tool rule259Model rankinglogistic regression284
From a pool of 3,353 customers who had not ordered in 90 days (29% of whom ordered again), 500 emails chosen by the model reach 284 returning buyers against 259 by recency and about 145 at random.
Score tenth 1526 customers92%Score tenth 2526 customers75%Score tenth 3526 customers60%Score tenth 4526 customers52%Score tenth 5526 customers43%Score tenth 6526 customers34%Score tenth 7525 customers28%Score tenth 8525 customers23%Score tenth 9525 customers19%Score tenth 10525 customers10%Test base rate
Test window after September 2011. The top tenth reordered at 92%; the bottom tenth rarely did.
MethodAUC (95% CI)PR-AUC
Rule: bought in last 90 days0.688 (0.675–0.701)0.581
Rule: recency rank0.762 (0.749–0.775)0.685
Logistic regression0.789 (0.778–0.801)0.761
Gradient boosting0.761 (0.748–0.774)0.740

The model was chosen on an earlier window and tested once on a later one. Logistic regression beat recency ranking by 0.027 AUC (95% CI 0.018 to 0.035); gradient boosting lost to it and was not used.

On-site buying signals

Top 5% of sessions237 of 1,091 buyers22%Top 10% of sessions416 of 1,091 buyers38%Top 20% of sessions672 of 1,091 buyers62%Top 30% of sessions815 of 1,091 buyers75%
A separate session score on 12,330 store sessions: the top 10% of scored sessions hold 416 of the 1,091 test-period buying sessions. Data: UCI Online Shoppers Purchasing Intention, 12,330 sessions

Data and method

Orders
UCI Online Retail II (Chen, 2019), CC BY 4.0: 1,067,371 invoice lines, 5,942 customers, 43 countries, Dec 2009 to Dec 2011
Cleaning
19,494 cancellations, 34,335 duplicates and 5,991 service lines removed; 22.8% of rows have no customer ID and are left out of customer models
Split
Select on a window to 2011-03-12; test once on customers as at 2011-09-10, outcomes to 2011-12-09
Sessions
UCI Online Shoppers Purchasing Intention (Sakar et al., 2018), 12,330 sessions

Segments come from k-means on log recency, frequency and spend (four clusters; silhouette checked from two to eight). The 90-day model uses recency, frequency, spend, basket size, tenure and last-90-day activity.

Assumptions and limitations

  • Prediction is not persuasion: some of the 284 would have returned anyway. A hold-out group is needed to measure the lift from the email itself
  • The spend forecast ran far below actual in the test quarter (£1.46M against £2.81M, because of the pre-Christmas peak); it is used only to rank, where its top 10% still capture 60% of next-quarter revenue
  • 54% of buyers the model missed were one-time customers with a median 304 days since their order
  • A UK gift wholesaler selling to shops; a consumer store needs its own refit

What a store would do next

  • Week 1: export 24 months of Shopify orders and customers
  • Week 2: segment, score and check against last quarter; agree messages per segment
  • Weeks 3 to 8: run the win-back flow with a 20% hold-out
  • Then: keep or stop on reorders and margin against the hold-out

Technical depth

Out-of-time testing matters here because seasonality is strong: a random split would mix November into training and testing. Guest checkouts are reported separately so the share of "known" customers is not overstated.

Analysis code (Python)

Financial impact

About A$18,900 a year in extra margin from the same send

Value is the extra returning buyers from the same number of win-back emails, at the store’s order value and margin.

InputValueSource
Win-back emails a month500
Returning buyers: model284Measured
Returning buyers: recency259Measured
Average order valueA$180
Gross margin35%

Calculation

  1. Extra buyers a month25
  2. Extra orders a year300
  3. Extra margin a yearA$18,900

Annual margin over recency targetingA$18,900

Some returning buyers would order without an email; a hold-out group measures the true lift before scaling up.

Implementation and rollout

Four weeks from Shopify export to segments and a win-back flow in Klaviyo

  1. Week 1

    Connect

    Pull two years of orders from Shopify or WooCommerce; clean refunds, duplicates and guest checkouts.

    Manaz

  2. Week 2

    Segment and score

    Build segments and the 90-day score; backtest against the current recency rule.

    Manaz

  3. Weeks 3 to 4

    Sync and send

    Segments and scores sync to Klaviyo or Mailchimp as tags; first win-back send with a 20% hold-out.

    Marketing coordinator

  4. Then

    Review

    Monthly refresh; keep the flow if the hold-out shows lift.

    Owner, Manaz

Tools

Shopify or WooCommerce orders, Klaviyo or Mailchimp tags and flows, Python logistic regression, refreshed monthly, Xero for margin by product

Team

  • Owner sets offers and discount limits
  • Marketing coordinator writes the messages and approves each send
  • Manaz pipeline, segments, score and monthly check

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Discounts sent to customers who would reorder anywayLoyal segment gets early access, not codes; hold-out measures lift.
Consent rules under the Spam Act 2003Only subscribed customers are tagged; unsubscribes sync back nightly.
Guest checkouts without a customer IDMatched on email where consented; otherwise left out and reported.
Seasonal peak distorts the scoreScore tested across the November peak; refitted each month.

Key takeaways

Key takeaways

  • Recency is a strong rule; the model adds a modest, measured gain.
  • Segments decide the message: loyal big spenders need contact, not discounts.
  • A simpler model beat gradient boosting on the later window and was kept.
  • The session score is a separate tool for remarketing, tested on its own data.