Three systems in use. Three times I went looking for the flaw myself.
Each of these shipped. Each one has a section where I attacked my own work and found something. That section is the reason to read it.
- 01Evaluation · Applied AI quality
Mapping Studio
811 of 813
automated quality checks passed on the live run(Measured)
A freight forwarder hand-wrote a translation rulebook for every trading partner it onboarded. Three specialist days each, and only a handful of people could write one. The agent drafts it now, and an analyst reviews one in sixteen minutes for about $3 of compute.
Then I tested the quality metric I had built to prove it worked. It could settle two arguments out of eighty-one, so this ships as a review console rather than an automated pipeline.
- My role
- Architecture, evaluation design and the analyst console. AI wrote the code.
- Status
- Deployed, in analyst use. Reviewed end to end by a senior analyst.
- Stack
- Python · FastAPI · Gemini on Vertex AI · SQLite
Read the case study
- 02Forward deployed · Solutions engineering
EDI Copilot
8 minutes
on a document that took thirty to fifty by hand(Measured)
Analysts check invoice files against a partner's two-hundred-page rulebook, five or six times a day. The files carry customer data the company is contractually bound to protect, so AI was off the table. The model runs inside a redaction zone now, behind a human sign-off. Thirty to fifty minutes by hand, eight with the tool.
Then I handed my own redacted file to a stronger model and asked it to name the customer. It did, off a single barcode. Barcodes are caught by checksum arithmetic now, not by appearance.
- My role
- Field research, redaction and refusal design, the whole system. AI wrote the code.
- Status
- Live pilot on a deployed server. Full rollout waits on the client.
- Stack
- Next.js · TypeScript · Gemini 2.5 Flash on Vertex AI · SQLite
Read the case study
- 03Agent engineering · Human-in-the-loop
Chalkline
7 human gates
specified points where a person has to approve before the machine may continue(Measured)
A school wanted its textbook lessons as short animated videos. Twelve stages turn a lesson PDF into a narrated 3D video for four to ten year olds, and a human has to sign off before the machine may carry on.
It read the lesson correctly and then made a video with none of it in. My automated check passed it, and so did a person, in seventy seconds. Both of those gates were redesigned because of it.
- My role
- Every architecture, evaluation, product and budget decision. I wrote none of the code by hand.
- Status
- In use at the school. Pro bono.
- Stack
- Python · LangGraph · Gemini
Read the case study
How I work
Five steps, and the case study where you can watch each one happen.
- 01
Intent modeling
I watch the work before I scope it. Shadowing the EDI analysts produced the two reframes the whole system rests on, and neither was in the brief.
EDI Copilot, discovery → - 02
Agent topology
I decide where the model sits and what it can reach. In EDI Copilot the model runs in one stage, inside a redaction zone, behind a human approval. In Mapping Studio three components are barred from calling a model at all, permanently, by rule.
Mapping Studio, the system → - 03
System prompts
The prompt is a contract, not a paragraph. Source of truth scoped to one document, a required citation field, and a named refusal verdict the model must use when that document is silent.
EDI Copilot, refusal → - 04
Evaluation design
I build the measurement before I trust the output, then check whether the measurement can adjudicate anything. In Mapping Studio it could not, and that finding changed the product.
Mapping Studio, evaluation → - 05
Observability
Every model call logged with tokens, latency, cost and the job that issued it. Every approval and override written to an audit trail with a hash of exactly what was sent.
Mapping Studio, cost and observability →
The demo is about five percent of the actual job.
Anyone can get an agent to work once, on a clean input, in a screenshot. The part that decides whether it survives contact with a real company is the other ninety five percent: the guardrails, the evaluation, the human checkpoints, the audit trail, and the change management that gets an analyst to trust the thing.
That work is invisible until it is missing. My job on a project is to name it early enough that it lands on the plan instead of in the post mortem.
- Deterministic
Deterministic by default
The model gets invoked where judgment is genuinely irreducible and nowhere else. In Mapping Studio the compilers, the path builder and the renderer make zero model calls, permanently, by rule.
Mapping Studio, the boundary
- Human
Checkpoints by cost asymmetry
Human in the loop is a design decision, not exception handling. Put the person where a mistake is still cheap to fix.
Chalkline, both gates passed it
- Verified
Refuse before you guess
An agent that always answers will eventually answer wrong with confidence. needs_business_review is a first-class verdict, and a pass has to cite the passage that permits it.
EDI Copilot, refusal