The long version.

01 · The domain, properly

Why this is not a matching problem

Three documents get collapsed by everyone outside the field. A schema says what can be sent. A guideline says what must be sent and what each field means. A mapping says how one company's format translates into another's. Automating the first is straightforward. The third is a judgement about meaning.

The example that made it concrete. The partner sends a date. The receiving format has several date fields that could plausibly take it. Which one is correct depends on a code attached to a completely different part of the message, and if that code is absent, the correct answer changes again. Column alignment does not produce that. Neither does string similarity.

The structure compounds it. In the reference rulebook for the live lane, 20 of 49 fields in scope carry more than one correct answer, up to 16 answers for a single field, each right under different conditions[M](Measured).

So a mapping is not a list of field pairs. It is a set of conditional rules per field, and the conditions carry most of the information.

What the pipeline does, end to end

Two properties worth calling out. One code path, not a demo path and a real path: the web console routes to the same compilers and the same aligner the command line invokes. And crash resilience by construction: every job persists the moment it completes, so a run killed at job 100 of 133 keeps 100 jobs of work and resumes.

02 · The evaluation, in full

What I built

Deterministic parsers, no LLM-as-judge anywhere in any primary measurement[M](Measured).

Eight scoring buckets, so a failure has a type rather than a score: CORRECT, NO_MAPPING_AGREE, PARTIAL, WRONG_SOURCE, MISSING, SPURIOUS, EXCLUDED_GROUP_LEVEL, OUT_OF_GOLD_SCOPE. Two lenses: strict literal comparison, and a substance lens that symmetrically strips tautological scoping conditions from both sides before comparing.

What it reported

Five full EDIFACT baselines[M](Measured):

The pre-registered exit bar, written down before the runs: untouched-row rate at or above 0.50, wrong-source rate exactly 0, unrepairable condition-grammar failures exactly 0. On the final complete like-for-like comparison it was met on 0 of 3 terms[M](Measured).

XML/JSON lane, first full inbound baseline: 6.8% strict, 13.0% substance[M](Measured).

Finding one: my scorer had a bug that reported zero

The first implementation keyed gold by target node in a dictionary. Because 20 of 49 fields carry multiple gold rules, that dictionary silently collapsed them to one arbitrary row by last-write-wins, and reported a false 0% / 0%[M](Measured).

I found it, rewrote scoring to evaluate every individual gold row, and rescored offline from already-collected proposals at $0.00 additional spend[M](Measured).

The quieter lesson: I could rescore historical runs offline and for free because every proposal had been persisted with full provenance. I had not stored it for that reason. I stored it because keeping the full record costs nothing and discarding it is irreversible.

Finding two: the denominator was structurally unsatisfiable

The generation design proposes one answer per field. The reference data carries up to 16 answers per field, discriminated by conditions[M](Measured). One answer can never satisfy all sixteen. The metric was penalising a shape mismatch between the output contract and the reference contract, not an error.

I recorded it as a design finding for a future rule-DSL runtime rather than adjusting the metric. A test you have quietly reshaped to fit your own output is not a test.

Finding three: the reference data could not adjudicate

Category breakdown of the first 42: supports model A 0, supports model B 1, supports both 7, supports neither 5, silent 29[M](Measured).

In roughly 97.5% of real disagreements the reference specification had nothing to say. Not because it was poor work, but because a mapping specification records one expert's defensible choice on one day and does not enumerate the alternatives that expert rejected.

So "disagrees with the reference" cannot be read as "wrong." The accuracy metric was measuring stylistic convergence with one analyst. Every figure in the baseline table above is a real measurement of the wrong thing.

What I replaced it with

Five deterministic oracles. No reference mapping, no model, no expert. 811 of 813 passed[M](Measured).

I report O4 as vacuous rather than dropping it, and O5 as 54 of 56 rather than rounding up. A set of oracles curated for presentation is the same failure as a metric reshaped to fit the output.

I also split the buckets honestly. One baseline reported 90 SPURIOUS rows; most were regions the reference document had never covered, so the system had answered a question the reference was never asked. Those moved to OUT_OF_GOLD_SCOPE and out of scoring. Same data, honest denominator, uglier bucket counts, more truthful metric.

Abstention as a first-class output

The AI's instructions carry one rule that is the whole thesis, written as a format requirement:

On the largest production run, of 583 rules covering 575 fields: 204 carried a mapping, 379 declared no match with a reason, and nothing was silently dropped[M](Measured). Confidence split 200 high, 383 medium.

One decision deliberately left open

The faster model became the live default under a written rule, with a pre-registered rollback trigger that remains armed pending human adjudication of 123 disagreement rows[M](Measured). It has not been disarmed because the evidence that would justify disarming it does not exist yet.

03 · The agent contract

System prompt architecture · cached prefix plus per-job payload

Code enforcement is half of the boundary. The other half is stated to the model, so the two cannot drift apart.

Four of the nine ground rules restate the boundary in the model's own language: never invent a field, element or code not in the evidence; the receiving structure is fixed; targets are IDs taken exactly from part F, so the model never writes a path string; strict structured output only.

Two more are the ones I would point an evaluation team at:

• If ambiguous, record the runner-up at lower confidence. Disagreement is data. Discarding the second-best answer discards the only evidence the first one was close.
• Never mark a proposal high-confidence without stating the evidence.

04 · Failure analysis

Rate limits were not my fault, and I tested that rather than assuming it

Early runs lost 17 of 88 jobs to HTTP 429[M](Measured). The tempting diagnosis is your own concurrency. At concurrency 1, with zero contention from my own side, 5 of 30 jobs, 16.7%, still hit 429[M](Measured). External and bursty.

Mitigation: a shared token bucket smoothing submissions to a measured 6 requests a minute, a bounded worker pool, and a sweep-and-retry pass recovering 429-failed jobs at the end of a run rather than losing them. Result on the full production run: 133 of 133 jobs, 0 not attempted[M](Measured).

Runaway generation, and a counterintuitive correlate

Worst observed: one job at 252.2 s, another riding the full 65,536-token ceiling for 227.3 s[M](Measured). Job size did not correlate. The three largest jobs in the plan consistently completed fast and clean.

The actual correlate was conditional complexity[M](Measured): an either-or rule spanning two source sections, or a qualifier judgement grounded in absent or corrupted evidence. The model was not overloaded. It was stuck, and had no way to notice.

C = M × 1.25, rounded up to the next 1024, floor 8192
no brake applied if C ≥ 32768
where M = the largest successful output that model has ever produced

Plus a one-shot uncapped retry, so a legitimately long answer can never be permanently lost to the brake. Post-fix, four ceiling-slam attempts each died in 64.2 to 83.0 s and self-recovered[M](Measured). Worst case fell from 252.2 s to 83.0 s.

Degenerate output, and calibrating a detector against real data

One instruction returned a 27-character unit repeated to 13,239 characters, with a stray NUL byte that crashed the Excel writer[M](Measured). Two layers: at generation, detect a 1 to 64 character unit repeating ten or more times and re-roll once; at render, strip illegal control characters and truncate any cell at 2,000 characters.

The threshold matters more than the mechanism. The longest legitimate instruction anywhere in that run was 77 characters, one outlier in 421 rows[M](Measured), so the detector cannot plausibly fire on real content. I wrote negative tests proving six ordinary short constants are never flagged.

Optimisations that were real on a probe and false at scale

The thinking budget turned out to be a soft target the model routinely exceeded, not a hard ceiling: a 95% win on six jobs, an 11% win on thirty[M](Measured). I ran that cycle four times before writing it down as a rule.

05 · Cost engineering

Fixed credit budget, hard rule that it would never touch paid billing. Budget ceiling checked before the transport layer, per-run and cumulative caps configurable per model, a mid-run tripwire, a cooperative stop that never interrupts an in-flight job but persists every remaining one with a visible cancelled_by_user reason, and a complete per-call ledger.

The full outbound production run, from the run database[M](Measured)

Sum of call latencies 5,010.81 s against 1,708 s wall clock implies effective concurrency 2.93 against a configured 3. The scheduler was doing what it was told.

A statistical bug in my own estimator

The estimator projected a run's total from a median. Total spend is N × mean, and this workload is heavily right-skewed: mean $0.0287170 against median $0.009358, a 3.07× skew[M](Measured). Validated with a holdout: fit on one real run, predict a different one.

Wrong in the direction that would have quietly permitted an overspend. Both figures are pinned by a test-suite assertion so the regression cannot recur silently.

Context caching, stated precisely

The project's own presentation material cites "97.5% of input tokens served from context cache" as a general engine statistic. Computed from the run databases[M](Measured):

The 97.78% is real and lane-specific, reflecting an EDIFACT prompt architecture built around a 128,367-token shared cached prefix across 88 job occurrences. The production lane uses per-job payloads and does not benefit.

06 · The review surface, in detail

An error-message standard, written down

That last clause is a silent-failure guard: node count, section count, resolved root and schema version are shown before a run can start, so a misconfiguration that compiles to the wrong thing is visible before it costs money.

07 · Delivery governance

• Briefs are self-contained. Anything discussed only in chat is void.
• Contracts are locked. The implementer must stop and say so rather than route around a constraint that blocks it.
• A written definition of done. Test suite passes with real output pasted into the report, state file updated, ledger entry recorded, commit, push. Every task.
• Measurement wins. Any fact asserted in a brief must be computed from the repository before being written down. One brief of mine asserted 9 unresolvable types. The reachability walk proved 8. The record says 8.
• Every figure in a report must appear verbatim in the report. A file path pointing at a number is not a substitute for the number.
• An append-only ledger. 205 numbered rulings including superseded ones, marked superseded rather than deleted. History you can rewrite is not evidence.
• Cost tripwires and stop rules registered before a run, not after.

Scale, all[M](Measured). 947 tests passing in 368.91 s, offline, zero network, zero spend · 22,701 lines of application Python, 47,275 including tests and tooling · 205 rulings · 13 validators · 119 commits over 26 calendar days · about $34 to $35 a month recurring infrastructure · deployed on a single VM behind an HTTPS reverse proxy with a written runbook and additive-only schema migrations.

08 · What I'd do differently

Build the reference-free oracles first. Weeks on reference-based scoring before discovering it could settle 2 of 81. The oracles were cheaper, needed no expert data, and were the metrics that held. I had the ordering exactly backwards.

Get a specialist in front of it in week one. Two sessions changed the interface more than every decision I made alone, and they came late.

Stop trusting small probes sooner. Four cycles before I wrote the rule down. Two would have done.

Freeze earlier. Freezing the EDIFACT lane at 0 of 3 exit terms was right. I held on past the point the evidence justified.

Instrument the review itself. The system logs every model call and every cent. It logs analyst verdicts. It never logged review duration, which is why the review figure is client-reported rather than measured. A small change that would have made the headline number fully mine.

09 · Published limits

• 01No measured acceptance rate yet. The overlay design makes one computable the moment enough reviews land. Anyone quoting an acceptance percentage should be able to show you the denominator.
• 02Reference-based accuracy is not a metric I trust on this problem. The figures in section 02 are published because they are what the instrument reported.
• 03The outbound direction is unverified. Full production run complete, in analyst validation. The ledger records it as UNVERIFIED, not passed.
• 04Version 1 covers X12 4010 and 5010, EDIFACT D96A, D01B and D10B, XML and JSON, roughly 95% of observed partner transactions[C](Client-stated). CSV, fixed-length and proprietary formats are out of scope.
• 05The client and the integration partner are unnamed by agreement.