All case studies
Case study
Chalkline
Subject
AI lesson videos for a school
File
03 of 03
Description of goods

The system read the lesson correctly, then made a video with none of it in.

My automated quality check approved it. So did a human, in seventy seconds.

A private English-medium school in India wanted its textbook lessons as short animated videos. I built a system that takes a lesson PDF and returns a narrated 3D video: agents read the lesson, plan what to teach and in what order, write the script, turn the textbook’s flat drawings into 3D character references, then generate and stitch the clips. Twelve stages, seven mandatory points where a human has to approve before the machine may continue. Then I audited one run end to end, and what I found is the reason this page exists.

Client
A private English-medium school in India
The job
Textbook lessons as short animated videos
My role
Every architecture, evaluation, product and budget decision
Stages
Twelve
Today
In use at the school, pro bono
Coverage
0 of 7
Lesson items the system found, then left out of the finished video.
Measured
The video
11 of 12
Beats spent retelling the poem, none on the exercises.
Human check
70 seconds
The human approval that let it through.
Act 1 of 5

Benchmark

Answer key: I benchmarked the agent against a video I had made by hand, not against a rubric.

My first attempt at an agentic video system failed on visual inconsistency. So I built the next one manually, one shot at a time, and made it the answer key: 8 shots, each split into three timestamped blocks, each block carrying six named fields and four mandatory audio tracks. The live system was producing 28 to 31 flat shots with none of that structure.

Result

A structural gap, not a polish gap. A rubric would have scored it fine.

Figure A· Diagram

Hand-built golden set

8 shots

10 seconds each, built by hand one shot at a time

  • 3 timestamped blocks per shot
  • 6 named fields per block
  • 4 mandatory audio sub-tracks

What the live system was producing

28 to 31 shots

flat micro-shots per video

No internal timestamped beats, no audio direction, no continuity bridges, and generic style language.

The answer key on the left is a video I made by hand, and the structure inside it is what the measurement is made of. The right-hand column is not a worse version of it. It has none of it.
Act 2 of 5

Audit

The run: I audited one run end to end, and it had thrown the lesson away.

The lesson plan named five sub-topics. Stage 1 found and correctly tagged all seven of the lesson’s exercises: the fill-in-the-blank pairs, prepositions of place, the ascending-order sets, the spelling correction, the crossword, the listening activity, the cultural extension. Then the planner spent all 300 seconds of runtime retelling the poem.

Result

7 of 7 items understood, 0 of 7 delivered. Comprehension was never the problem.

Figure B· Diagram
Each exercise in the lesson: found by stage 1, and whether it reached the finished video
Found by stage 1In the finished video
Fill-in-the-blank pairsYesNo
Prepositions of placeYesNo
Ascending-order setsYesNo
Spelling correctionYesNo
CrosswordYesNo
Listening activityYesNo
Cultural extensionYesNo
7 of 70 of 7
The lesson’s seven exercises. The first stage found and tagged every one; the finished video taught none of them.
Figure C· Diagram
Where the twelve beats of the finished video went, one block per beat.

The gates: Both my quality gates passed it, one automated, one human.

The critic ran a full redraft cycle, two calls to it and two back to the writer, then approved the same plan again. The human gate cleared the script in 1m10s. Neither was holding a list of what the lesson contained, so neither could see what was missing.

Result

I had been treating “the human approved it” as proof of correctness. It proves nothing without a reference to check against.

Figure D· Diagram
  1. The automated critic

    Approved

    After a full redraft cycle: two calls to it, two back to the writer.

  2. The human gate

    Approved

    The script cleared in 1m10s.

A list of what the lesson contained

Held by neither gate, so neither could see what was missing.

Two gates, both passed, and the one thing neither was given.
Act 3 of 5

Fix

Coverage: I banned silent dropping.

Every activity in the lesson brief must now either be assigned to a beat or explicitly marked with a reason for being skipped. Prepositions of place was a named concept with zero beats and the critic approved the plan anyway, so the critic must now fail exactly that case.

Result

The failure moves from invisible to logged. Silence is indistinguishable from correctness until somebody audits by hand, and in this project’s history that happened once.

Critics: I split coverage from craft, and gave each its own critic.

A critic named “quality” had quietly narrowed to mean “narrative quality,” and about the narrative it was right every time. I stopped widening it and specified a second single-purpose critic that checks one thing: did every concept get taught. Two independent passes, never one blended score.

Result

A run can now fail for the reason it actually failed. A blended score had been hiding the only axis that mattered.

Act 4 of 5

Cut

Scope: I cut the one capability I could not govern, and never opened a code editor.

Lip-sync came out of the spec entirely, 13 stages down to 12. Narration runs as external voiceover instead: cheaper, removes a whole class of drift between clips, and hands me control of script and pronunciation. Spend is checked against a run-level cap before calls go out, not after.

Result

$0.58 against a $5.00 cap on the audited run. Stages 1 to 6 only. Video generation spend was never captured, so I am not quoting a per-video figure.

Figure E· Diagram

Spend on the audited runin US dollars

  • The audited run: $0.58
  • Run-level cap: $5.00
The audited run’s spend against its cap, stages 1 to 6. Video generation spend was never captured, so it is not on this ruler.
Act 5 of 5

Change

Changes: What I changed because of it.

What I changed because of it: before the audit, and after
Before the auditAfter
Coverage checkNone. The critic scored narrative qualityA dedicated critic that fails any concept with zero beats
Dropping an activityPermitted, and silentAssigned to a beat, or marked with a reason
What the human sees at the script gateThe scriptThe script beside the list of what the lesson contained

There is no after-measurement on this page. I found the failure, specified the fix and made it the project’s top priority. A fix is not a result, and I am not going to present it as one.

Declaration

How these numbers were measured.

  • Every figure on this page is measured. None is projected.

What this does not claim.

  • One lesson, one audit. Everything here comes from a single end-to-end review. No held-out set, no repeat runs, no regression suite. Anyone quoting a pass rate off this should be able to show you the denominator, and I cannot.
  • No before and after. No baseline exists for what a lesson video cost or took at this school beforehand, so there is no saving to quote and I have not invented one.
  • The coverage fix is a specification. It was the top priority six weeks after the audit and I have not published a measurement of it working.
  • The school and its founder are unnamed by agreement.
Next case study · 01Mapping Studio The first version’s output was good, and the analysts went quiet. I rebuilt it to feel like the software they use all day, and they started relying on it.