All work

EDI Copilot

Redaction design · Agent refusal · Supply chain EDI

A compliance check that took up to fifty minutes, done in eight.

EDI Copilot

Then I handed my own redacted file to a stronger model and asked it to name the customer. It did.

Up to 6× faster
30 to 50 minutes by hand, 8 with the tool(Client-stated and measured)
2,375 values masked, 0 sent
swapped out before the prompt exists(Measured)
6 of 8 stages call no model
by rule, not by prompt(Measured)
Every large retailer publishes its own rulebook for how a partner's invoice files must look, sometimes two hundred pages(Client-stated). An analyst checks one file against one rulebook, line by line, five or six times a day. Seventy analysts do it(Client-stated). Those files carry names and account numbers the company is contractually bound to protect, which is why nobody could use AI on them.
Status
A live pilot on a deployed server. Full rollout waits on client approval and resource allocation.
My role
Every architecture, redaction, evaluation and product decision. I wrote none of the code by hand.

The six decisions

  1. 01

    I decided what the AI is never allowed to see, before I built anything.

    A leaked customer name here is not a bug, it is a breached contract. So I made exposure structurally impossible rather than unlikely. Every sensitive value in both files becomes a token before the prompt is assembled, and six of the eight stages never call a model at all.

    Result: 7,724 differences analysed, 2,375 confidential values masked, none of them sent(Measured).

The boundary · where a model may run
  1. UploadHuman
  2. ParseNo model call
  3. CompareNo model call
  4. MaskBoth documents are tokenized before the prompt is assembledNo model call
  5. ApproveNothing is sent until a person approves the exact payloadHuman, mandatory
  6. AnalyzeOne prompt contract, rows in batches of twentyThe only model stage
  7. UnmaskLocally, against a mapping the model never seesNo model call
  8. ReviewHuman

Barred, permanently, by rule

  • Delimiter detection
  • Parsing
  • Difference classification
  • Barcode validity

No model anywhere in these. Barcode validity is checksum arithmetic. Unmasking runs locally against a mapping the model never sees.

The eight stages and what runs each one. Six make no model call at all, one is a human gate, one calls the model. The stage-by-stage detail is in the long version.
  1. 02

    I masked by pattern first, and it broke my own model's input.

    My first pass hid anything shaped like a code. "or" is Oregon, "in" is Indiana, so both vanished from the rulebook's own sentences, taking the structure the model needed with them. The verdicts got worse. I now mask by position in the record, never by how a string looks.

    Result: the last commit in the entire product repository is that fix(Measured). Over-masking is not caution, it is a quieter way to be wrong.

  2. 03

    I attacked my own redaction with a stronger model, and it won.

    Barcodes look like ordinary numbers, so my first pass left them alone. One product barcode can re-identify an entire anonymous file. I only found that by attacking my own work, and the attack won.

    Result: barcodes are now caught by checksum arithmetic, not by appearance(Measured). I ran it late because I was confident, which is the exact condition under which you should run it early.

One segment, three states
  1. Raw, on the hostStays on the host
    N1STMEGA MART STORES

    The file as it arrives. This state never leaves the building.

  2. Masked, as the model sees itSent to the model
    N1ST[PARTY_NAME_1]

    Tokenized before the prompt is assembled, and the same value gets the same token across the whole document rather than only where it changed.

  3. Restored, after the answer comes backStays on the host
    N1STMEGA MART STORES

    Unmasked locally, against a mapping the model never sees.

Element 0 is the qualifier saying which party this is. It carries no identity and stays readable, or the model loses the structure it needs. Element 1 is the party name and goes. The name shown is a synthetic test fixture from the product repository; the two further elements on a real segment carry the party identifier and are left off here rather than invented.
  1. 04

    I made silence produce a refusal instead of a guess.

    The dangerous verdict is not a wrong one. It is a confident pass on a rule nobody checked. So a pass only stands if it quotes the passage permitting it, and code downgrades any pass it cannot verify.

    Result: three categories of difference are settled before the model is ever called(Measured). This sends more rows to a person than a looser system would. That is the price of never turning silence into false certainty.

The verdicts, and the only way the code may move one
  1. valid_per_guidelinesexplicitly permitted, and here is the passage
  2. no_action_requiredstructural or mechanical, settled without the model
  3. clarification_requireda specific question would settle it
  4. rejectthe rulebook forbids this, and here is the passage
  5. needs_business_reviewthe rulebook is silent, so a human decides An uncited pass is downgraded to here

The downgrade only ever moves toward caution. reject and clarification_required are never touched, and nothing is ever moved up the ladder.

Five verdicts. The one that matters most is the one that declines to answer. Anything not positively sanctioned goes to a human.
  1. 05

    I deleted the confidence score I had already shipped.

    The first result screen showed a verdict and a confidence label. Watching analysts work, "medium" changed nobody's behaviour. It performed nuance while sending the reviewer back into the spec to find the rule anyway, so the screen had added a step, not removed one. I cut it, and two other controls with it.

    Result: three shipped features removed(Measured). Calibration is provenance, not confidence. Show the receipt, not the score.

Shipped, then removed

  • The confidence label

    Confidence: mediumRemoved.

    Nobody could act differently on it, and it sat beside an already probabilistic verdict.

  • The verdict override

    Override verdict ▾Removed.

    Read as a duplicate of the reviewer disposition control beside it.

  • The per-row verdict badge

    BREAKS THE RULESRemoved.

    A top-level count, not a marker that belongs on every row.

What replaced the first of these is provenance: the exact passage, the page it came from, the verbatim prompt, and a fingerprint of what was sent.
  1. 06

    I wrote none of the code, then audited it as though a stranger had.

    A planning model turns one decision at a time into a brief; a coding agent executes it. Then I audited the result as if it were someone else's. It caught my own README claiming a safety feature that did not exist, and an audit trail whose fingerprint did not match what had been sent.

    Result: both fixed, out of 12 briefs and one working day of commits(Measured). The scarce skill is the governance, not the code.

What changed

By hand(Client-stated)With the system
One document30 to 50 minutes8 minutes(Measured)
Confidential datakept AI out entirelymasked on the host, never sent(Measured)
A pass verdictthe analyst's memory of a 200-page rulebooka quoted passage, or no pass(Measured)

3,000 hours a year handed back(Projected), on the client's own planning volume. A second projection runs 17× higher(Projected) on assumed volume alone. It is in the long version, because quietly picking the bigger number is how figures like these stop being believed.

What this does not claim

  • There is no accuracy measurement, and I will not imply one. No answer key, no precision, no recall. Five human overrides across 7,724 differences(Measured) is a usage signal, not a quality score. Anyone quoting it as accuracy should be asked for the answer key. I do not have one.
  • One model stage, not one model call. Rows go out in batches of twenty(Measured). An earlier draft of mine said "exactly one call." That was wrong, and it sounded better than the truth.
  • The leak scan is a check, not a guarantee. A heuristic tuned for one transaction set, and its badge says so.
  • The pilot counts are one snapshot, dated 26 June 2026(Measured).

The long version · about 15 minutes

The full redaction design, the refusal policy, the audit that caught my own false claim, and what I still cannot measure. Where I would want an evaluation team to spend their time.

Read the long version

Next case study

Chalkline

It read the lesson correctly and then made a video with none of it in. My automated check passed it, and so did a person, in seventy seconds. Both of those gates were redesigned because of it.

Read it

All three, side by side