How the hard parts work.

Five decisions from systems I shipped, drawn as schematics. Each shows a quiet failure mode in data sync, money, real-time state, or generated content, plus the rule that contains it.

I traced each exhibit back to production code. The highlighted invariant is the condition that must remain true.

erp-sync --keyset-drain

§V · FIG.01

FIG. 01

No-silent-loss ERP sync

The hard part: the ERP’s change-cursor is a minute-resolution timestamp, so a burst of >1000 rows in one minute would silently strand the overflow.

watermark = MINUTE-resolution timestamp>1000 rows in one minute → overflow stranded silently

composite keyset cursor(timestamp, primary-key)
drain a saturated minutepage-by-page
restart at a 1-minute overlap floorre-reads in-place edits
idempotent upsert

Never advance the watermark unless accepted + rejected == rows_sent. Silent drops are caught, not skipped.

payouts --exactly-once

§V · FIG.02

FIG. 02

Exactly-once payouts

The hard part: a payout API call can time out after the money has already moved. “No response” is not the same as “it failed.”

provably rejected

DEFINITIVE failuremoney never moved

RELEASE fundssafe to re-claim

might have succeeded

AMBIGUOUSkeep PENDING

reconcilewebhook · status-check · admin

resolvepaid|failed

Release funds only on a provable failure. A maybe-succeeded transfer is never double-paid.

realtime --server-authoritative

§V · FIG.03

FIG. 03

Cheat-resistant real-time6 → 100+ players

The hard part: in a live multiplayer round, the client cannot be trusted. So the correct answer can never reach it, and the score cannot be set by it.

token

client requests join
gateauth + payment + room membership
scoped tokensubscribe / presence only

the correct answer is never sent to clients during a live question

answer

answer submit
atomic compare-and-setfirst-answer-wins guard
server-authoritative scoring
broadcast6 → 100+ subscribers

round

round advance
coordinator electiondeterministic leader + liveness takeover
idempotent phase claim

Scoring is server-authoritative and answers are never leaked, so you cannot fake a correct answer. Scales 6-player rooms → 100+ broadcast.

locks --one-global-order

§V · FIG.04

FIG. 04

Deadlock-free money pathsLogistics · Postgres

The hard part: two money flows that lock the same rows in opposite orders will eventually deadlock. Postgres aborts one with 40P01.

flow Abillspayments

flow Bpaymentsbills

each holds what the other wants → 40P01

billssorted by bill number
then the payment row

unlocked discovery readlock in global orderrevalidate under lock

A single global lock ordering across every money path, so the cycle that caused 40P01 cannot form.

questions --initial-rollout

§V · FIG.05

FIG. 05

First production AI pipeline30 people → 2

The hard part: a model will produce a confidently wrong exam question, and a wrong question does not look wrong. It looks like a question, right up until it is graded against a student.

generate

versioned promptchanges are traceable to output
batch jobbulk, not per-question calls
four providersautomatic fallback on failure

one provider going down does not stop the content team

check

draft question
schema validationshape, options, answer key
repair passmalformed items go back, not forward

publish

initial review gatemandatory in the first rollout
live to students

first reduced to 2 reviewers; the current workflow has no routine manual handling

In the first production rollout, no generated question reached a student without approval. That boundary exposed the failure modes before later automation removed the routine manual step.

These are the load-bearing parts, drawn small. The full case studies, with context, stack and what shipped, live in ~/work.