Case 01 — Exactly-once at scale
Exactly-once reward dispatch across a multi-pod cluster
Situation
A customer-engagement platform dispatches rewards for campaigns reaching up to 300K recipients each — roughly 1.5M dispatches at peak across 5 concurrent campaigns, processed by multiple pods in parallel.
Constraint
Message reordering and pod races could double-pay or drop rewards. Real money; retries are not free; a pod can die mid-dispatch at any moment.
Approach
Designed an epoch-gate + compare-and-swap (CAS) idempotent dispatch, backed by a 4-pass mark-first reconciliation orchestrator so interrupted campaigns self-heal without duplicate or lost sends.
Outcome
Exactly-once delivery held in production at full campaign scale — including through mid-dispatch pod deaths.
300K recipients / campaign · ~1.5M dispatches at peak · 0 duplicates