All case studies
Case study
EDI Copilot
Subject
AI for a supply chain compliance check
File
02 of 03
Description of goods

A compliance check that took up to fifty minutes, done in eight.

Then I handed my own redacted file to a stronger model and asked it to name the customer. It did.

Every large retailer publishes its own rulebook for how a partner’s invoice files must look, sometimes two hundred pages. An analyst checks one file against one rulebook, line by line, five or six times a day. Seventy analysts do it. Those files carry names and account numbers the company is contractually bound to protect, which is why nobody could use AI on them.

The job
Checking invoice files against a retailer’s rulebook
The team
Seventy analysts, five or six files a day each
My role
Every architecture, redaction, evaluation and product decision
A rulebook
Sometimes 200 pages
Today
A live pilot on a deployed server
Time
30 to 50 minutes replaced by 8 minutes
Up to 6× faster, on one document.
Measured
Confidential data
2,375 masked, 0 sent
Swapped out before the prompt exists.
Model use
6 of 8 stages
Call no model at all, by rule, not by prompt.
Act 1 of 5

Protect

Boundary: I decided what the AI is never allowed to see, before I built anything.

A leaked customer name here is not a bug, it is a breached contract. So I made exposure structurally impossible rather than unlikely. Every sensitive value in both files becomes a token before the prompt is assembled, and six of the eight stages never call a model at all.

Result

7,724 differences analysed, 2,375 confidential values masked, none of them sent.

Figure A· Diagram
  1. Upload: Human
  2. Parse: No model call
  3. Compare: No model call
  4. Mask: No model callBoth documents are tokenized before the prompt is assembled
  5. Approve: Human, mandatoryNothing is sent until a person approves the exact payload
  6. Analyze: The only model stageOne prompt contract, rows in batches of twenty
  7. Unmask: No model callLocally, against a mapping the model never sees
  8. Review: Human

Barred, permanently, by rule

  • Delimiter detection
  • Parsing
  • Difference classification
  • Barcode validity

No model anywhere in these. Barcode validity is checksum arithmetic. Unmasking runs locally against a mapping the model never sees.

The eight stages and what runs each one. Six make no model call at all, one is a human gate, one calls the model.
Act 2 of 5

Break

Masking: I masked by pattern first, and it broke my own model’s input.

My first pass hid anything shaped like a code. “or” is Oregon, “in” is Indiana, so both vanished from the rulebook’s own sentences, taking the structure the model needed with them. The verdicts got worse. I now mask by position in the record, never by how a string looks.

Result

The last commit in the entire product repository is that fix. Over-masking is not caution, it is a quieter way to be wrong.

Attack: I attacked my own redaction with a stronger model, and it won.

Barcodes look like ordinary numbers, so my first pass left them alone. One product barcode can re-identify an entire anonymous file. I only found that by attacking my own work, and the attack won.

Result

Barcodes are now caught by checksum arithmetic, not by appearance. I ran it late because I was confident, which is the exact condition under which you should run it early.

Figure B· Diagram
  1. Raw, on the host

    Stays on the host

    N1STMEGA MART STORES

    The file as it arrives. This state never leaves the building.

  2. Masked, as the model sees it

    Sent to the model

    N1ST[PARTY_NAME_1]

    Tokenized before the prompt is assembled, and the same value gets the same token across the whole document rather than only where it changed.

  3. Restored, after the answer comes back

    Stays on the host

    N1STMEGA MART STORES

    Unmasked locally, against a mapping the model never sees.

Element 0 is the qualifier saying which party this is. It carries no identity and stays readable, or the model loses the structure it needs. Element 1 is the party name and goes. The name shown is a synthetic test fixture from the product repository; the two further elements on a real segment carry the party identifier and are left off here rather than invented.
Act 3 of 5

Refuse

Refusal: I made silence produce a refusal instead of a guess.

The dangerous verdict is not a wrong one. It is a confident pass on a rule nobody checked. So a pass only stands if it quotes the passage permitting it, and code downgrades any pass it cannot verify.

Result

Three categories of difference are settled before the model is ever called. This sends more rows to a person than a looser system would. That is the price of never turning silence into false certainty.

Figure C· Diagram
  1. valid_per_guidelinesexplicitly permitted, and here is the passage
  2. no_action_requiredstructural or mechanical, settled without the model
  3. clarification_requireda specific question would settle it
  4. rejectthe rulebook forbids this, and here is the passage
  5. needs_business_reviewthe rulebook is silent, so a human decidesAn uncited pass is downgraded here

The downgrade only ever moves toward caution. reject and clarification_required are never touched, and nothing is ever moved up the ladder.

Five verdicts. The one that matters most is the one that declines to answer. Anything not positively sanctioned goes to a human.

Confidence: I deleted the confidence score I had already shipped.

The first result screen showed a verdict and a confidence label. Watching analysts work, “medium” changed nobody’s behaviour. It performed nuance while sending the reviewer back into the spec to find the rule anyway, so the screen had added a step, not removed one. I cut it, and two other controls with it.

Result

Three shipped features removed. Calibration is provenance, not confidence. Show the receipt, not the score.

Figure D· Diagram
  • The confidence label

    Removed: Confidence: medium

    Nobody could act differently on it, and it sat beside an already probabilistic verdict.

  • The verdict override

    Removed: Override verdict

    Read as a duplicate of the reviewer disposition control beside it.

  • The per-row verdict badge

    Removed: BREAKS THE RULES

    A top-level count, not a marker that belongs on every row.

What replaced the first of these is provenance: the exact passage, the page it came from, the verbatim prompt, and a fingerprint of what was sent.
Act 4 of 5

Audit

Audit: I wrote none of the code, then audited it as though a stranger had.

A planning model turns one decision at a time into a brief; a coding agent executes it. Then I audited the result as if it were someone else’s. It caught my own README claiming a safety feature that did not exist, and an audit trail whose fingerprint did not match what had been sent.

Result

Both fixed, out of 12 briefs and one working day of commits. The scarce skill is the governance, not the code.

Act 5 of 5

Pilot

Value: What changed.

What changed: by hand, and with the system
By handWith the system
One document30 to 50 minutes8 minutes
Confidential dataKept AI out entirelyMasked on the host, never sent
A pass verdictThe analyst’s memory of a 200-page rulebookA quoted passage, or no pass

3,000 hours a year handed backProjected, on the client’s own planning volume. A second projection runs 17× higherProjected on assumed volume alone. I quote the smaller one, because quietly picking the bigger number is how figures like these stop being believed.

Declaration

How these numbers were measured.

  • 30 to 50 minutes, seventy analysts and the 200-page rulebook: the client’s own figures.
  • 8 minutes, 7,724 differences, 2,375 masked values, six of eight stages and five overrides: measured.
  • 3,000 hours a year and 17× higher: projections, labelled where they appear.

What this does not claim.

  • There is no accuracy measurement, and I will not imply one. No answer key, no precision, no recall. Five human overrides across 7,724 differences is a usage signal, not a quality score. Anyone quoting it as accuracy should be asked for the answer key. I do not have one.
  • One model stage, not one model call. Rows go out in batches of twenty. An earlier draft of mine said “exactly one call.” That was wrong, and it sounded better than the truth.
  • The leak scan is a check, not a guarantee. A heuristic tuned for one transaction set, and its badge says so.
  • The pilot counts are one snapshot, dated 26 June 2026.
Next case study · 03Chalkline It read the lesson correctly and then made a video with none of it in. My automated check passed it, and so did a person, in seventy seconds. Both of those gates were redesigned because of it.