- Case study
- EDI Copilot
- Subject
- AI for a supply chain compliance check
- File
- 02 of 03
A compliance check that took up to fifty minutes, done in eight.
Then I handed my own redacted file to a stronger model and asked it to name the customer. It did.
Every large retailer publishes its own rulebook for how a partner’s invoice files must look, sometimes two hundred pages. An analyst checks one file against one rulebook, line by line, five or six times a day. Seventy analysts do it. Those files carry names and account numbers the company is contractually bound to protect, which is why nobody could use AI on them.
- The job
- Checking invoice files against a retailer’s rulebook
- The team
- Seventy analysts, five or six files a day each
- My role
- Every architecture, redaction, evaluation and product decision
- A rulebook
- Sometimes 200 pages
- Today
- A live pilot on a deployed server
- Time
- 30 to 50 minutes replaced by 8 minutes
- Up to 6× faster, on one document.
- Measured
- Confidential data
- 2,375 masked, 0 sent
- Swapped out before the prompt exists.
- Model use
- 6 of 8 stages
- Call no model at all, by rule, not by prompt.
Protect
Boundary: I decided what the AI is never allowed to see, before I built anything.
A leaked customer name here is not a bug, it is a breached contract. So I made exposure structurally impossible rather than unlikely. Every sensitive value in both files becomes a token before the prompt is assembled, and six of the eight stages never call a model at all.
Result
7,724 differences analysed, 2,375 confidential values masked, none of them sent.
- Upload: Human
- Parse: No model call
- Compare: No model call
- Mask: No model callBoth documents are tokenized before the prompt is assembled
- Approve: Human, mandatoryNothing is sent until a person approves the exact payload
- Analyze: The only model stageOne prompt contract, rows in batches of twenty
- Unmask: No model callLocally, against a mapping the model never sees
- Review: Human
Break
Masking: I masked by pattern first, and it broke my own model’s input.
My first pass hid anything shaped like a code. “or” is Oregon, “in” is Indiana, so both vanished from the rulebook’s own sentences, taking the structure the model needed with them. The verdicts got worse. I now mask by position in the record, never by how a string looks.
Result
The last commit in the entire product repository is that fix. Over-masking is not caution, it is a quieter way to be wrong.
Attack: I attacked my own redaction with a stronger model, and it won.
Barcodes look like ordinary numbers, so my first pass left them alone. One product barcode can re-identify an entire anonymous file. I only found that by attacking my own work, and the attack won.
Result
Barcodes are now caught by checksum arithmetic, not by appearance. I ran it late because I was confident, which is the exact condition under which you should run it early.
Raw, on the host
Stays on the host
N1*ST*MEGA MART STORES
The file as it arrives. This state never leaves the building.
Masked, as the model sees it
Sent to the model
N1*ST*[PARTY_NAME_1]
Tokenized before the prompt is assembled, and the same value gets the same token across the whole document rather than only where it changed.
Restored, after the answer comes back
Stays on the host
N1*ST*MEGA MART STORES
Unmasked locally, against a mapping the model never sees.
Refuse
Refusal: I made silence produce a refusal instead of a guess.
The dangerous verdict is not a wrong one. It is a confident pass on a rule nobody checked. So a pass only stands if it quotes the passage permitting it, and code downgrades any pass it cannot verify.
Result
Three categories of difference are settled before the model is ever called. This sends more rows to a person than a looser system would. That is the price of never turning silence into false certainty.
valid_per_guidelinesexplicitly permitted, and here is the passageno_action_requiredstructural or mechanical, settled without the modelclarification_requireda specific question would settle itrejectthe rulebook forbids this, and here is the passageneeds_business_reviewthe rulebook is silent, so a human decidesAn uncited pass is downgraded here
The downgrade only ever moves toward caution. reject and clarification_required are never touched, and nothing is ever moved up the ladder.
Confidence: I deleted the confidence score I had already shipped.
The first result screen showed a verdict and a confidence label. Watching analysts work, “medium” changed nobody’s behaviour. It performed nuance while sending the reviewer back into the spec to find the rule anyway, so the screen had added a step, not removed one. I cut it, and two other controls with it.
Result
Three shipped features removed. Calibration is provenance, not confidence. Show the receipt, not the score.
The confidence label
Removed:
Confidence: mediumNobody could act differently on it, and it sat beside an already probabilistic verdict.
The verdict override
Removed:
Override verdictRead as a duplicate of the reviewer disposition control beside it.
The per-row verdict badge
Removed:
BREAKS THE RULESA top-level count, not a marker that belongs on every row.
Audit
Audit: I wrote none of the code, then audited it as though a stranger had.
A planning model turns one decision at a time into a brief; a coding agent executes it. Then I audited the result as if it were someone else’s. It caught my own README claiming a safety feature that did not exist, and an audit trail whose fingerprint did not match what had been sent.
Result
Both fixed, out of 12 briefs and one working day of commits. The scarce skill is the governance, not the code.
Pilot
Value: What changed.
| By hand | With the system | |
|---|---|---|
| One document | 30 to 50 minutes | 8 minutes |
| Confidential data | Kept AI out entirely | Masked on the host, never sent |
| A pass verdict | The analyst’s memory of a 200-page rulebook | A quoted passage, or no pass |
3,000 hours a year handed backProjected, on the client’s own planning volume. A second projection runs 17× higherProjected on assumed volume alone. I quote the smaller one, because quietly picking the bigger number is how figures like these stop being believed.
How these numbers were measured.
- 30 to 50 minutes, seventy analysts and the 200-page rulebook: the client’s own figures.
- 8 minutes, 7,724 differences, 2,375 masked values, six of eight stages and five overrides: measured.
- 3,000 hours a year and 17× higher: projections, labelled where they appear.
What this does not claim.
- There is no accuracy measurement, and I will not imply one. No answer key, no precision, no recall. Five human overrides across 7,724 differences is a usage signal, not a quality score. Anyone quoting it as accuracy should be asked for the answer key. I do not have one.
- One model stage, not one model call. Rows go out in batches of twenty. An earlier draft of mine said “exactly one call.” That was wrong, and it sounded better than the truth.
- The leak scan is a check, not a guarantee. A heuristic tuned for one transaction set, and its badge says so.
- The pilot counts are one snapshot, dated 26 June 2026.