The long version.
Methods and mechanisms. The decisions and outcomes are on the main page. This is the document for someone who has already decided the main page was interesting and now wants to know whether the work underneath it is real.
A note on evidence quality, up front. This project has no repository, no test suite and no run database available to me now. The measured figures below were read off one real run report produced by one real end-to-end run. That is the strongest evidence class this project has, and it is weaker than the evidence behind my other two case studies. I would rather say that here than have you work it out later.
How to read the numbers
Every number on this page carries one of three marks.
- [M]
- Measured.
- [C]
- Client-stated.
- [P]
- Projected.
Where provenance is weak, it says so in the sentence rather than in a footnote.
01 · What the system is
A private English-medium school in India teaches from national curriculum textbooks. The school wanted its lessons as short animated videos. The system takes a lesson PDF and returns a narrated 3D animated video.
Twelve stages. Originally thirteen. Each stage is specified in the same eight-part shape: purpose, input, sub-steps, agent boundaries, output, tools, hard parts, validation gate. Every sub-agent inside a stage is specified in the same seven-field shape: input, output, output format, tools and models, model tier, hand-off target, and whether it runs sequentially or in parallel.
That uniformity is not tidiness. A stage described in a different shape from its neighbours is a stage whose boundaries nobody has actually decided.
The spine of the run is the Lesson Brief: a version-locked, schema-validated JSON artifact produced by Stage 1 that every downstream stage consumes. Stages do not talk to each other in prose. They talk through a validated contract, or they do not talk.
Model tiering is deliberate and cost-aware. Three tiers: a small model for parsing and classification, a medium model for general reasoning, and a top-tier model reserved for high-stakes pedagogical reasoning. Most stages do not need the expensive model, and paying for it everywhere is how a per-video cost quietly triples.
02 · The golden set, and why it exists
Gloss: a golden set is a hand-made reference output that you trust, which you then measure the machine against. It is an answer key.
I had already tried to build an agentic video system once before this project. It failed on visual inconsistency: characters would not hold their appearance across shots. I stopped and built a video by hand instead, one shot at a time, through a manual chain of research, image generation, video generation and editing tools, screenshotting the last frame of each clip to seed the next one for continuity.
That hand-made video became the benchmark. It matters that it was validated by real reception from the school's founder, its board and parents, rather than by me deciding it was good.
Its structure, which is the part that carries the measurement:
| Level | What it holds |
|---|---|
| 8 shots, 10 seconds each(Measured) | 80 seconds of finished video(Projected) |
| A storybeat per shot | 2 to 3 sentences on the emotional and pedagogical purpose of that shot(Measured) |
| 3 timestamped blocks per shot | [00:00–00:03], [00:03–00:07], [00:07–00:10](Measured) |
| 6 named fields per block | Cinematography, Subject, Action, Context, Style and Ambiance, Audio(Measured) |
| 4 mandatory audio sub-tracks | Dialogue, SFX, Ambient, Music(Measured) |
What the live agentic system was producing instead: 28 to 31 flat micro-shots per video, with no internal timestamped beats, no audio direction, no continuity bridges and generic style language(Measured).
Both numbers are mine. That comparison is the single most useful measurement in the project, and the reason is that the reference had structure. Eight structured shots against 28 flat ones is a diagnosis. A rubric scoring "visual quality: good" is not.
Structural fidelity to the golden set is treated as a necessary but not sufficient condition for output quality. Necessary, because a video without audio direction cannot be good by accident. Not sufficient, because section 4 of this document is what happens when you meet every structural condition and still fail the lesson.
03 · Where the AI is not allowed to act
The claim this portfolio is built on is that I design what the AI must not do. Here is the list for this system, with the reason attached to each.
| Boundary | Why |
|---|---|
| Lip-sync removed entirely, 13 stages to 12 | An entire generative capability cut because its risk and cost outran its payoff. Narration runs as external voiceover, which is cheaper, removes cross-clip audio drift, and gives full control of script and pronunciation. |
| Narration is never the video model's native voice | Same reason, plus a specific failure mode: native narration does not carry cleanly across clip-extension hops if it is absent from the last second of the source clip. |
| Stage 2.4 tags diagrams, it does not redraw them | Redrawing happens later, in a different stage. Tagging and rendering are different concerns and a model that does both will quietly do neither well. |
| Stage 10 overlay is programmatic compositing, not generation | Low technical risk by construction. There is no reason to let a model near text that must be exactly correct. |
| Per-clip QC runs deterministically | File integrity, expected duration and gross artifact detection, checked mechanically with ffprobe and a single-frame vision check before any human or model spends attention on the clip. |
| Final stitching is a deterministic pass | A unifying colour-grade over the assembled video, to counter drift that generative steps introduce across shots. |
| Never hardcode lesson-specific proper nouns into a system prompt | Character names change every run. System prompts encode consistency principles; the per-run data carries the names. This was my own binding correction after I found it happening. |
| The budget cap is checked before a call goes out | Spend is governance of somebody else's money, not an engineering afterthought. |
04 · The audit
One lesson. One end-to-end run. A Grade 5 English poetry lesson, chosen because it was representative rather than because it was easy.
4.1 What the run produced
A roughly five-minute video, visually strong. Characters held their appearance scene to scene. Last-frame continuity chaining worked, with no jarring scene jumps. On every dimension I had built a check for, it passed.
4.2 What it left out
The lesson plan named five sub-topics: story understanding, family relationships, vocabulary development, comprehension, and moral values.
Stage 1 correctly identified and tagged seven distinct exercise items. Here is the full table, which is the strongest single piece of evidence in this project:
| Lesson item | Found by Stage 1 | Present in the video |
|---|---|---|
| Fill-in-the-blank pairs, 7 examples with images | Yes | No |
| Prepositions of place | Yes | No |
| Ascending-order exercises, 5 sets | Yes | No |
| Spelling correction, 10 words | Yes | No |
| Crossword puzzle, antonyms | Yes | No |
| Homophone listening activity | Yes | No |
| Cultural extension, words for father | Yes | No |
7 of 7 found. 0 of 7 delivered(Measured).
The planner had 300 seconds of runtime to allocate. It produced 12 beats: 11 poem narration, 1 reflection, 0 exercise(Measured). The grammar concept, the vocabulary drill, every exercise and the lesson's own stated moral-values outcome were all absent from the finished video, after being correctly understood upstream.
The source document also records this as "roughly 30% of lesson content survived." I am not publishing that figure. It is an approximate audit judgment, its derivation appears nowhere, and the seven-row table above does not arithmetically produce it. The itemised table is measured. The percentage is not, and a number that cannot be reconstructed does not belong on a page whose whole argument is that numbers should be reconstructable.
4.3 The failures that were not mine
Two things in the audit failed for reasons outside the system's design, and I separate them deliberately rather than absorbing them into the headline.
- 854×480 delivered against a 1080p request(Measured). The video vendor was silently downscaling. Not my system, and I cannot take credit for finding a fix because there wasn't one to make.
- The run finished but never closed. Status still showed
in_progressat 8m26s, with two configured agents never invoked(Measured).
4.4 The telemetry failure, which was mine
Seven image generation calls, costing $0.28, were reported with a duration of 0.0 seconds(Measured). The cost ledger was right and the timing instrument was wrong. The lesson I take from it is narrow and worth stating: the instrument you use to audit a system needs its own audit, because a metrics pipeline that is wrong in a boring way trains you to ignore it when it is wrong in an interesting way.
05 · Why both gates passed it
This is the section I would want an evaluation team to read.
5.1 The automated critic
The critic ran. It was not skipped, it was not misconfigured, it did not error. It made two calls and sent the plan back to the writer twice: a full redraft cycle(Measured). Then it approved the same plan again.
My diagnosis: the critic was nominally responsible for "quality," and had quietly narrowed to mean "narrative quality." Judged on the narrative, it was right every single time. The video was a good retelling of the poem. Nothing in the critic's definition of its own job required it to ask whether the lesson had been taught.
5.2 The human gate
The script gate is mandatory. A human has to approve before the machine may continue. It was cleared in 1m10s(Measured).
The obvious reading is that the reviewer was careless. I think that reading is wrong, and that it is the more comfortable of the two available conclusions. The reviewer was shown the script. The script was good. Nothing on the screen was incorrect. There was no list of what the lesson contained, and therefore no way to notice an absence.
An omission is invisible without a reference. You cannot see what is not there by looking harder at what is. I designed a gate that asked a human to detect a category of failure that the gate's own interface made undetectable, and I recorded "treating the human approved it as proof of correctness" as a named anti-pattern of this project as a result.
This is the part I got wrong. Not the critic, which is an ordinary scoping mistake. The human gate was wrong at the design level: I had built the ritual of oversight without the evidence that makes oversight possible, and then counted it as a control.
5.3 The seven gates, and what each blocks
Seven mandatory human decision points were specified, non-configurable by design. Every gate offers a three-way decision, never a bare yes or no: Approve, Reject or Adjust at the early gates, Approve, Regenerate or Drop at the generation gates. A gate that can only say yes is not a gate, it is a speed bump with a log line.
The earliest gate sits at the end of lesson understanding, where the reviewer sees detected age band, content summary, sample illustrations and all flags. It is placed there because age-appropriateness and child safety must be settled before any downstream spend happens, and because everything after it is more expensive to redo.
Three of the seven cluster around the video generation stage, which is the expensive one. Gate density tracks cost and irreversibility, not stage count.
A discrepancy I am not going to paper over: the two gates that actually fired in the audited run, the plan lock and the script lock, do not appear in the canonical seven-gate list in the specification. The implemented gate set and the specified gate set are not the same, and the source material never reconciles them. I can tell you seven gates were specified. I cannot tell you seven shipped.
06 · The fix
Written the same day as the audit, and made the project's top priority on the grounds that nothing else meaningfully improves the product until pedagogical completeness is enforced.
1. Silent dropping becomes illegal. Every activity in the lesson brief must either be assigned to a beat or explicitly marked as skipped, with a reason. The reason field is mandatory, because an optional one gets left blank and an absent signal is exactly the failure being fixed.
2. The critic must fail zero-coverage concepts. Any plan where a named concept maps to zero beats fails. In the audited run, prepositions of place was a named concept with zero beats and the critic approved the plan anyway. The rule is written against the exact case that got through.
3. Coverage and craft get separate critics. A new single-purpose coverage critic, rather than widening the existing one. Two independent passes, never one blended score. A blended score is how an axis that fails gets averaged away by axes that pass, which is precisely what happened: visually excellent, pedagogically empty, overall fine.
4. The human gate gets a reference. The script gate shows the script beside the list of what the lesson contained. Same reviewer, same 70 seconds, different question: not "is this good" but "what is missing."
Status: specified, and recorded as not yet implemented as of the last status check in the source. Six weeks after the audit it was still the top priority and still unbuilt. I am not going to call a specification a result.
07 · Delivery
I wrote none of this by hand. That is the constraint the whole delivery method is built around.
The working shape: short turns, frequent checkpoints, confirm before proceeding, no over-generation. That last one is a rule I imposed after redirecting the implementing agent away from long, dense, unconfirmed responses, and it became the project default. A long unconfirmed response is not progress. It is a larger thing to throw away.
The stop-and-confirm gate on the implementing agent itself: read the repository, report your understanding, propose the map, and wait for confirmation before building anything. The same human-in-the-loop discipline I designed into the product, pointed at the tool building the product.
Cost governance: a cost tracker accumulates per-call spend into a run-level ledger and raises alerts against a hard cap. The audited run spent $0.58 against a $5.00 cap(Measured), covering stages 1 through 6. Video generation spend was not captured in the run report, so there is no per-video figure here and I have not estimated one.
Spec discipline: the version-two revision cut the lip-sync stage and consolidated four redundant validation sub-steps into a single validator, on the grounds that they were all confirming the same thing rather than each refining it. Removing checks from a system is usually a bad instinct. Removing four checks that all assert the same predicate is housekeeping.
08 · The second pipeline
A second video generation pipeline was designed against a different model, to sit behind a model selector alongside the working one.
Two decisions worth stating. First, the existing working pipeline stays untouched. A second option is not an excuse to destabilise the thing that currently produces output. Second, the prompt logic was re-authored from scratch rather than ported, because the two models differ in per-call clip length and extension behaviour, and those limits shape architecture rather than just wording. Treating models as drop-in replacements for each other is how you inherit an architecture designed for someone else's constraints.
This pipeline was designed and not built. Any claim about its behaviour would be a claim about a system that does not exist, so there are none on this page. In particular, a source note that a four-part splitting approach "measurably reduces character drift" carries no measurement anywhere and is excluded.
09 · What is not measured
The honest inventory, in one place.
- n = 1. One lesson, one audit, one run report. No held-out set, no repeat runs, no regression suite. The evaluation design in this project is better than the evaluation volume.
- No client baseline. There is no
[C]number anywhere in this project. Nobody recorded what a lesson video cost or took at this school before the system existed, so there is no saving, no multiple and no payback period to quote. - No outcome measurement. Four success metrics were defined with methods and none have results. No classroom pilot result is recorded in the source.
- The adoption assumption is untested. Whether teachers actually use these videos in class is the assumption the entire project rests on, and it was never tested in the material available to me.
- The visual fidelity target of roughly 85% recognizability is a design target with no stated measurement method. It appears nowhere as a result.
- Reproducibility is a known gap, named as such in the source material itself.
10 · What I would do differently
Define "complete" before defining "good." Every check I built measured whether the output was good. Not one measured whether it was complete. Those are different questions and I built eleven ways to ask the first one.
Give every gate an artifact to compare against, or delete the gate. A human confirmation with no reference is not a control. It generates a log entry that looks like assurance and functions as noise, and it is worse than no gate because it stops anyone looking harder.
Audit earlier and smaller. One end-to-end audit found a failure that had been present the whole time. It was expensive enough that it was performed exactly once in the project's history, which is the real reason the failure survived so long. A cheaper, narrower coverage check, run every time, would have caught it in the first week.
Name critics for what they check, never for a virtue. "Quality" is not a specification. "Every named concept maps to at least one beat" is.
Back to the short version