Manaz Afzal

Manufacturing · Quality and yield

Catching failing lots before final test, on production the model never saw

A risk screen across 446 process signals for a high-throughput line, tested in time order, with the weekly drift checks that keep it reliable.

53%of failing lots caught inspecting 30% of lots
29%lower cost than inspecting everything, at a 20-to-1 cost ratio
74%of signals shifted between fitting and test weeks

Executive summary

A lot-level risk screen across 446 process signals catches 53% of failing lots by inspecting 30%, on later production it never saw.

53%of failing lots caught inspecting 30%
29%lower cost than inspecting everything, at a 20-to-1 cost ratio
74%of signals shifted between fitting and test weeks

Situation

A high-throughput line logs hundreds of process signals per lot; failing lots are found only at final test, after the value has been added.

Approach

A risk screen fitted in time order, chosen on validation weeks and tested on 314 later lots, against the best single signal and against inspecting everything.

Result

AUC 0.65 on later lots; 53% of failures caught at 30% inspection, against 24% for the best single signal. 74% of signals drifted, so the screen is refitted weekly.

The problem

A semiconductor or electronics line logs hundreds of process signals for every lot, and a minority of lots fail final test after the value has been added. Fabs use virtual metrology and yield prediction to decide which lots to inspect early, but accuracy figures quoted on shuffled data rarely hold on next month’s production.

I wanted a reading a plant manager can trust: on later production the model never saw, how many failing lots a risk screen catches for a given inspection budget, and what drift checks keep it reliable.

Team
Process engineering, quality and yield engineering, production planning
Scale
High-throughput lines where each lot carries significant added value
Systems
MES such as Siemens Opcenter or Critical Manufacturing, historians such as AVEVA PI, SPC tools
Measure
Failing lots caught per hundred inspected; cost of inspection plus escaped failures

What I built

A risk screen that scores each lot from its process readings as it leaves the line, before final test. The highest-risk lots go to an extra in-line inspection or an engineer’s review; the rest flow on. Because process signals drift, the screen is refitted every week on the latest labelled lots and checked for shift before each release.

Process readingsSignals per lotfrom the historianRisk scoreRanked againstthis week’s lotsExtra inspectionTop band reviewedbefore final testWeekly refitLatest test resultsand drift check
The drift check compares each signal’s recent distribution with the fitting weeks and holds the release if many have moved.

Results

Lots were kept in time order. The method was chosen on validation weeks, refitted on everything before 2008-10-02, and tested on the 314 lots that followed, of which 17 failed. On those lots the screen ranks a failing lot above a passing one 65% of the time (AUC 0.65, 95% range 0.52 to 0.78).

0%25%50%75%100%0%20%40%60%80%100%Best single sensor (chosen on history)Screen on all sensorsRandom orderInspect 30%:53% of failures caughtShare of lots inspected, highest risk first
Inspecting the riskiest 30% of lots catches 53% of failures. The single sensor that best separated failures in the fitting weeks catches 24%, no better than random on later lots. Data: UCI SECOM
Lots inspectedFailures caught, screenFailures caught, best single sensor
10%24%12%
20%24%24%
30%53%24%

What it is worth

With an escaped failure costing 20 inspections, inspecting every lot in the test fortnight costs the equivalent of 314 inspections and inspecting none 340. The screen, inspecting its riskiest lots, costs 223: about 29% below inspecting everything. When an escape costs 50 inspections, the cheapest plan with the screen inspects most lots and still costs 10% less than inspecting everything.

Cost of an escaped failure, in inspectionsInspect nothingInspect everythingBest single sensorScreen on all sensors
10×170314163156
20×340314296223
50×850314306283

Why it has to be refitted

Between the fitting weeks and the test fortnight, 74% of the usable signals shifted by a statistically clear margin. The sensors the screen leans on most moved as well. A fixed rule on one or two signals, set once by a vendor or a project team, degrades within weeks.

Sensor 59importance 3.10.24Sensor 519importance 1.10.34Sensor 247importance 1.00.34Sensor 33importance 0.80.12Sensor 65importance 0.80.13Sensor 290importance 0.80.15Sensor 460importance 0.80.06Sensor 477importance 0.70.12
A KS statistic of 0.3 means the recent and historic readings differ substantially; several of the most useful signals are in that range.

The failure rate itself moves week to week, so the inspection band is set as a share of each week’s lots.

0%5%10%15%20%25%07-2008-0308-1708-3109-1409-2810-12Weekly failure rate
Random forest, all sensors0.60Gradient boosting, all sensors0.53Logistic, stronger penalty0.46Regularised logistic regression0.43Logistic, weaker penalty0.41coin toss
Linear models fell below a coin toss on the validation weeks; tree ensembles held up. The random forest was chosen there, before the test fortnight was scored.

Data and method

Data
UCI SECOM: 1,567 production lots from a semiconductor line, July to October 2008, with 590 process signals and a pass or fail result at in-house line testing; signals are anonymised
Cleaning
Signals with one value or more than half missing removed, leaving 446; median imputation
Split
Fitted on the first 60% of lots by time, methods chosen on the next 20%, refitted on 80% and tested on lots from 2008-10-02 to 2008-10-17
Methods
Random forest, gradient boosting and regularised logistic regression; best single signal chosen on history as the baseline
Measures
AUC with lot-level bootstrap range, failures caught at fixed inspection shares, total cost at stated cost ratios, KS drift per signal

Limitations

How I would take it live

Financial impact

About 29% off the combined cost of inspection and escaped failures

The gain comes from inspecting the riskiest lots instead of every lot, while catching about half the failures before final test. Lot counts and costs are assumptions.

InputValueSource
Lots a month2,000 high-throughput line
Cost of an extra inspectionA$400
Escaped failure cost20 inspections scrap and rework
Cost reduction against inspecting all, test fortnight29%Measured

Calculation

  1. Inspect every lot, monthlyA$800,000
  2. Screened inspection plus escaped failuresA$568,153
  3. Monthly savingA$231,847

Annual savingA$2,782,166

Engineering time saved by pointing reviews at drifting signals is a further gain and is not counted here.

Implementation and rollout

A quarter in shadow, then live routing

  1. Weeks 1 to 3

    Extract and fit

    Lot-level signals from the historian and final-test results from the MES; time-ordered fitting and validation.

    Process engineer, Manaz

  2. Week 4

    Set the band

    Inspection share agreed from real scrap, rework and inspection costs.

    Quality manager, production planning

  3. Weeks 5 to 16

    Shadow quarter

    Scores published per lot with a weekly drift report; no routing yet.

    Yield engineer, Manaz

  4. Week 17 onward

    Live routing

    Top band to extra inspection; weekly refit; every escape reviewed.

    Quality team, Manaz on support

Tools

MES such as Siemens Opcenter or Critical Manufacturing, AVEVA PI or another process historian, Python, scikit-learn, Power BI or Grafana drift dashboard

Team

  • Quality or yield manager owns the inspection band
  • Process engineers act on drift alerts and reason codes
  • Manaz screen, refit pipeline and drift monitoring

The timeline is a plan for a typical engagement; it depends on data access and how many systems are involved.

Risks and mitigations

What could go wrong in production, and the control for each

RiskMitigation
Few failures in any one weekShadow quarter before the band is fixed; ranges reported with every figure.
Signals driftWeekly refit and a drift check that holds release when many signals move.
Engineers distrust a black boxReason codes name the signals behind each high score.
Costs changeBand reviewed quarterly against actual scrap and inspection costs.

Key takeaways

Key takeaways

  • On later lots the screen catches 53% of failures inspecting 30%.
  • The single best signal from history is no better than random a month later.
  • 74% of signals drifted, so weekly refits are part of the design.
  • When escapes are very costly the screen inspects most lots, and the cost table shows where that line sits.