Manufacturing · Quality and yield
Catching failing lots before final test, on production the model never saw
A risk screen across 446 process signals for a high-throughput line, tested in time order, with the weekly drift checks that keep it reliable.
Executive summary
A lot-level risk screen across 446 process signals catches 53% of failing lots by inspecting 30%, on later production it never saw.
Situation
A high-throughput line logs hundreds of process signals per lot; failing lots are found only at final test, after the value has been added.
Approach
A risk screen fitted in time order, chosen on validation weeks and tested on 314 later lots, against the best single signal and against inspecting everything.
Result
AUC 0.65 on later lots; 53% of failures caught at 30% inspection, against 24% for the best single signal. 74% of signals drifted, so the screen is refitted weekly.
The problem
A semiconductor or electronics line logs hundreds of process signals for every lot, and a minority of lots fail final test after the value has been added. Fabs use virtual metrology and yield prediction to decide which lots to inspect early, but accuracy figures quoted on shuffled data rarely hold on next month’s production.
I wanted a reading a plant manager can trust: on later production the model never saw, how many failing lots a risk screen catches for a given inspection budget, and what drift checks keep it reliable.
- Team
- Process engineering, quality and yield engineering, production planning
- Scale
- High-throughput lines where each lot carries significant added value
- Systems
- MES such as Siemens Opcenter or Critical Manufacturing, historians such as AVEVA PI, SPC tools
- Measure
- Failing lots caught per hundred inspected; cost of inspection plus escaped failures
What I built
A risk screen that scores each lot from its process readings as it leaves the line, before final test. The highest-risk lots go to an extra in-line inspection or an engineer’s review; the rest flow on. Because process signals drift, the screen is refitted every week on the latest labelled lots and checked for shift before each release.
Results
Lots were kept in time order. The method was chosen on validation weeks, refitted on everything before 2008-10-02, and tested on the 314 lots that followed, of which 17 failed. On those lots the screen ranks a failing lot above a passing one 65% of the time (AUC 0.65, 95% range 0.52 to 0.78).
| Lots inspected | Failures caught, screen | Failures caught, best single sensor |
|---|---|---|
| 10% | 24% | 12% |
| 20% | 24% | 24% |
| 30% | 53% | 24% |
What it is worth
With an escaped failure costing 20 inspections, inspecting every lot in the test fortnight costs the equivalent of 314 inspections and inspecting none 340. The screen, inspecting its riskiest lots, costs 223: about 29% below inspecting everything. When an escape costs 50 inspections, the cheapest plan with the screen inspects most lots and still costs 10% less than inspecting everything.
| Cost of an escaped failure, in inspections | Inspect nothing | Inspect everything | Best single sensor | Screen on all sensors |
|---|---|---|---|---|
| 10× | 170 | 314 | 163 | 156 |
| 20× | 340 | 314 | 296 | 223 |
| 50× | 850 | 314 | 306 | 283 |
Why it has to be refitted
Between the fitting weeks and the test fortnight, 74% of the usable signals shifted by a statistically clear margin. The sensors the screen leans on most moved as well. A fixed rule on one or two signals, set once by a vendor or a project team, degrades within weeks.
The failure rate itself moves week to week, so the inspection band is set as a share of each week’s lots.
Data and method
- Data
- UCI SECOM: 1,567 production lots from a semiconductor line, July to October 2008, with 590 process signals and a pass or fail result at in-house line testing; signals are anonymised
- Cleaning
- Signals with one value or more than half missing removed, leaving 446; median imputation
- Split
- Fitted on the first 60% of lots by time, methods chosen on the next 20%, refitted on 80% and tested on lots from 2008-10-02 to 2008-10-17
- Methods
- Random forest, gradient boosting and regularised logistic regression; best single signal chosen on history as the baseline
- Measures
- AUC with lot-level bootstrap range, failures caught at fixed inspection shares, total cost at stated cost ratios, KS drift per signal
Limitations
- The test fortnight holds 17 failures, so the range on every figure is wide. A live pilot runs for at least a quarter before the inspection band is fixed.
- Signals are anonymised, so the screen cannot say which tool or chamber is drifting; on a real line each signal is named in the historian and reason codes point engineers to it.
- The plant’s own scrap, rework and inspection costs set the real band.
- The data is from 2008. The method, time-ordered testing and drift checks carry over; the specific signals do not.
How I would take it live
- Extract lot-level signals from the historian and final-test results from the MES for the last six months
- Fit the screen with time-ordered validation and agree the inspection band from real scrap and inspection costs
- Run in shadow for a quarter, publishing the weekly drift check to process engineering
- Route the top band to extra inspection, refit weekly, and review every escaped failure with the yield engineer