Legal · Document automation
A review assistant that fills the contract checklist
Every checklist item comes back with the three passages most likely to answer it, so the reviewer reads about 6% of the contract.
Executive summary
For 94% of checklist items the right passage is in the assistant’s top three, so a reviewer reads about 6% of a contract per item instead of searching all of it.
Situation
Commercial teams review supplier and licence agreements against the same checklist. Judging a clause takes expertise; finding it takes reading, and keyword search misses half the time because contracts rarely use the checklist’s words.
Approach
Contracts split into passages; one text model per checklist item, trained on 408 lawyer-labelled CUAD contracts and scored on the 102 in the official test split.
Result
The assistant leads keyword search by 42 points (95% CI 38 to 45). It is weakest on rare clauses phrased many ways, such as liquidated damages (71%). 71 of 1,244 pairs still need finding by reading.
The business problem
A legal team reviewing commercial contracts works through a checklist: change of control, assignment, termination, liability caps. Firms using review tools such as Kira or Luminance report first-pass review times cut by a third to a half, because the tool finds the passages and the lawyer reads them.
I measured how often a review assistant puts the right passage for each checklist item in its top three, against keyword search, so the reviewer reads a small share of each contract.
- Team
- In-house counsel or a contracts manager, often one person
- Scale
- Tens to a few hundred agreements a month, many on the other side’s paper
- Systems
- SharePoint or Google Drive, Word, a checklist in Excel; sometimes Ironclad or LexisNexis
- Constraint
- Confidential documents stay in the firm’s tenancy; a lawyer signs off every item
What was built
The contract is split into passages. A text model trained on 408 lawyer-labelled contracts scores every passage for every checklist item. The review sheet lists the top three passages per item with a link back to the page; items with no likely passage are marked “not found” for the reviewer to confirm. The reviewer reads about 6% of the text per item instead of all of it.
Results
Where it struggles
| Weakest items (10+ test contracts) | Test contracts | Assistant top 3 | Keyword top 3 |
|---|---|---|---|
| Liquidated Damages | 14 | 71.4% | 50.0% |
| Competitive Restriction Exception | 16 | 75.0% | 43.8% |
| Volume Restriction | 17 | 82.4% | 0.0% |
| Ip Ownership Assignment | 23 | 82.6% | 52.2% |
| Post-Termination Services | 29 | 82.8% | 37.9% |
| Document Name | 102 | 84.3% | 23.5% |
71 of 1,244 pairs still need finding by reading. Rare clauses phrased in many ways miss most often.
Data and method
- Source
- CUAD v1 (Hendrycks, Burns, Chen and Ball, 2021): commercial contracts filed with the US SEC, labelled by lawyers
- Size
- 510 contracts, 41 clause types, 13,823 labelled spans, 53,001 passages
- Length
- Median 33,052 characters in 66 passages; longest 337,344
- Split
- CUAD official: 408 train, 102 test
Baselines: reading the first three passages (25%) and keyword search on each item’s name (53%). The assistant’s gain over keyword search is 42% points (95% CI 38% to 45%).
| Checklist item | Words the model leans on |
|---|---|
| Governing Law | governed, laws of, laws, the laws, governed by |
| Non-Compete | competitive, competitor, competing, competitors, compete |
| Termination For Convenience | terminate, termination, notice, written notice, any time |
| Cap On Liability | liability, damages, consequential, remedy, exceed |
| Change Of Control | of control, change of, change, control, merger |
| Anti-Assignment | assignment, assign, transfer, void, exhibit 16 |
Assumptions and limitations
- US public-company contracts; a firm’s own templates need a check on a few hundred of its own documents first
- The assistant finds passages; it does not judge whether terms are acceptable
- Scanned PDFs need OCR first, and OCR errors lower recall
- Rare items such as most-favoured-nation clauses have few training examples
What a team would do next
- Week 1: pick 200 to 300 past agreements and the checklist the team already uses
- Weeks 2 to 3: label or import existing review notes; refit and test on held-out agreements
- Weeks 4 to 8: reviewers use the sheet on live reviews and log misses
- Then: add items or playbook positions where reviewers ask for them
Technical depth
Passages are scored per item by logistic regression on TF-IDF word and two-word features, one model per item; results use the official split with no tuning on test contracts.