All work

Mapping Studio

Evaluation design · Agent boundaries · Logistics

A three-day expert job, done in sixteen minutes for 2% of the cost.

Mapping Studio

Then I tested the thing that was supposed to prove it worked, and found it couldn't tell right from wrong.

70× faster
One specialist now reviews 75 rulebooks in the time it took to write one(Projected)
98% cheaper
$200 of specialist time down to $3 of compute, per rulebook(Client-stated and measured)
811 of 813
automated quality checks passed on the live run(Measured)
A freight company hand-writes a translation rulebook every time it takes on a new trading partner. Three days of senior specialist time each, and only a handful of people in the company can write one, so every new customer waits in a queue behind their calendars. I turned writing those rulebooks into reviewing them.
Status
Deployed, in analyst use. Reviewed end to end by a senior analyst.
My role
Every product, architecture, evaluation and budget decision. I wrote none of the code by hand.

The six decisions

  1. 01

    I decided where the AI was not allowed to go, before I built anything.

    An invented field address that does not exist is the worst possible failure in this domain, so I made it structurally impossible rather than unlikely. Three parts of this system make zero AI calls, permanently, by rule. The AI proposes a field by name; code looks that name up and builds the address.

    Result: that entire category of failure cannot occur. The AI is trusted with meaning and never with addresses.

The boundary · where a model may run
  1. Compile the guideline documentsFour compilers normalise XSD, PDF, Word and JSON samples into one typed schemaNo model call
  2. PreflightRoot resolution and a reachability walk, before any run can startNo model call
  3. Plan the workRecursive descent over the target tree until each job fits a token ceilingNo model call
  4. AlignProposes a field by name. Never an address.The only AI stage
  5. ValidateThirteen validators, V1 to V13, before a human sees a rowNo model call
  6. ReviewSheet view and tree view, three-way verdict, edits stored as an overlayHuman
  7. ExportThe same data structure the UI renders, so screen and file cannot driftNo model call

Barred, permanently, by rule

  • The compilers
  • The path builder
  • The renderer

Zero AI calls, permanently, by rule. Enforced in code, not requested in a prompt.

The seven stages of the pipeline and what runs each one. Exactly one stage calls a model. The stage-by-stage detail is in the long version.
  1. 02

    I tested my own test, and it failed.

    Building an answer key is the easy half. I ran two AI models against each other, collected every case where they disagreed, and asked my expert answer key which one was right. It answered twice out of eighty-one(Measured).

    Result: my headline quality metric was measuring nothing, and I found that out before a customer did. Try it yourself below.

Interactive · real measured outcomesTried 0 · Settled 0

Two AI models disagreed 42 times. I had an expert answer key.

Click any tile to see what the answer key said about that disagreement. Try a few.

Pick a tile. The answer key gets one chance to tell me which model was right.

Every outcome above is the real measured distribution from the first pass of that test. No tile content is illustrative.

  1. 03

    I let the measurement change the product, not the other way round.

    If no automatic check can certify this work, then the deliverable cannot be an automatic export. I rebuilt the measurement around five checks that need no expert data at all, and rebuilt the product around a human holding the pen.

    Result: 811 of 813 checks passed on the live run(Measured). Both misses are empty container fields, not data.

Every check in the run, one dot each
811 automated checks passed2 failed
One dot per check on the live run. Both misses are empty container fields, not data. Reported as 54 of 56 on the mandatory-node check rather than rounded up.
  1. 04

    I made the tool resemble the one they already trusted.

    Two sessions with a real specialist changed the interface more than every decision I made alone. Dark panels read as "not the tool I know." I rebuilt the tree view to mirror the structure they already navigate, because their spatial memory of it is real expertise.

    Result: trust in a tool like this comes from resembling the incumbent, not from looking modern. That cut against my instincts, which is why it was worth having.

  2. 05

    I spent the client's money like it was scarce, because it was.

    The spending limit is checked before a request goes out, not after it comes back. A tripwire halts the run if spend crosses a threshold too early. I also caught my own cost estimate being 3.6× too low(Measured), in the direction that would have overspent their budget, and locked the corrected figures behind a test.

    Result: the largest run finished all 133 jobs, 25% under a cap I set before starting(Measured). A full year of running this costs about what four rulebooks used to cost by hand(Projected).

Same work, both costs, one scale
A hundred rulebooks, by handabout $20,000 a year
  • Senior specialist time

About 250 specialist days, from a handful of people who can do it.

A hundred rulebooks, with the systemabout $808 a year
  • Compute
  • Hosting, $34 a month

Anyone qualified to review a rulebook can produce one.

The by-hand figure is client-stated(Client-stated); compute and hosting are measured(Measured). Hosting is $34 a month, so a year of it is $408.
  1. 06

    I built the governance that let one person ship this in 26 days.

    I never opened a code editor. A planning model turns my decisions into one self-contained brief at a time; a coding agent executes exactly that brief. Every rule exists because something went wrong first: locked contracts, a written definition of done, an append-only log of 205 numbered decisions, and one standing rule that when a measurement disagrees with the plan, the measurement wins.

    Result: 22,701 lines of tested, deployed software and 947 passing tests, in 26 days(Measured). The scarce skill there is not the code. It is the system that made the code trustworthy.

What changed

By hand(Client-stated)With the system
One rulebook3 specialist days, about $20016 minutes, about $3(Measured)
Who can produce oneA handful of senior specialistsAnyone qualified to review one
A hundred a yearabout $20,000 and 250 specialist daysabout $400, plus $34 a month(Measured)

That is a full person-year of the company's scarcest people, handed back every year(Projected). It excludes rework avoided and revenue pulled forward by going live weeks earlier, both of which the client considers larger than the direct saving(Client-stated).

What this does not claim

  • No measured acceptance rate yet. The design makes one measurable the moment enough reviews land, and they have not landed. Anyone quoting an acceptance percentage should be able to show you the denominator. I cannot yet, so I am not quoting one.
  • Accuracy against an expert rulebook is not a measurement I trust on this problem. Section 02 is why. The reference figures are published in the long version because they are what the instrument reported, not because they measure quality.
  • One direction is still unverified. It completed a full production run and is with a specialist. The project log records it as unverified, not passed, because a run nobody has checked is not a run that passed.
  • The client is unnamed by agreement.

The long version · about 20 minutes

The full evaluation methodology, the five baselines, the cost engineering, the failure analysis and the delivery governance. It is where I would want an evaluation team to spend their time.

Read the long version

Next case study

EDI Copilot

Then I handed my own redacted file to a stronger model and asked it to name the customer. It did, off a single barcode. Barcodes are caught by checksum arithmetic now, not by appearance.

Read it

All three, side by side