How a 30-person content workflow became an automated pipeline

Producing the questions for one exam used to occupy a content team of about 30 people for a full day. The first production version reduced that to two reviewers working for under an hour, at higher volume. The workflow has since moved beyond that version and now runs without routine manual handling.

The problem was never "use AI." A model will happily produce a confidently wrong exam question, and a wrong question does not look wrong: it looks exactly like a question, right up until it is graded against a student who studied. The useful system had to increase output without turning model confidence into product correctness.

What the 30-person workflow was doing

The team was not only writing questions. It was moving each set through a chain:

1. choose the exam structure
2. draft questions manually
3. normalize difficulty and topic coverage
4. review formatting inconsistencies
5. rebuild the final paper into something the platform could ingest

Every handoff introduced delay. Even when the content quality was strong, the pipeline around it was slow.

The boundary around the model

I treated a model call as one step in the workflow, not as the workflow. The system coordinates:

• versioned prompts scoped to section and difficulty
• batch jobs across four model providers with automatic fallback
• schema validation for the question, options, and answer key
• a repair pass for malformed drafts
• explicit states for failed, malformed, repaired, and publishable output

Provider-specific logic stays at the edges. The rest of the pipeline works with one internal shape, so switching a model does not require rewriting generation, validation, and ingestion together.

The first rollout deliberately ended at a human approval gate. That was the right boundary while the team learned where failures clustered. It reduced the workflow from roughly 30 people to two reviewers without pretending that schema-valid meant correct. Later iterations removed the routine manual step; this article documents the first production boundary because it made the transition measurable and safe.

Failure had to become a visible state

Providers fail differently. One times out. Another follows the requested shape until question complexity increases. A retry can recover a transient failure or multiply latency around a bad prompt.

Those cases cannot collapse into one "generation failed" message. The workflow needs to distinguish:

• a provider failure that can fall back
• a malformed item that needs repair
• an item that satisfies the pipeline's publishability checks
• an item that must not move forward

That state model matters more than the fallback loop itself. It lets one provider go down without stopping the workflow, while preventing malformed output from silently moving forward.

What changed

The initial system moved a full-day, 30-person process to two reviewers handling 100+ questions in under an hour. That made the failure modes visible enough to improve the pipeline further; the current workflow no longer depends on routine manual handling.

If I rebuilt it now, I would add evaluation datasets earlier and make provider and validation failures easier to compare in the operator dashboard. The model layer will keep changing. The observable quality boundary should not.

The wider product context is in the AfterBoards and AfterGrad case study. The same pipeline is drawn alongside the other production invariants in How the hard parts work.