Rewards to 300,000 people at once, exactly once

How exactly-once works at this scale

For engineers: the stack, the dispatch path, the decisions behind it, and the code shape of the core mechanism.

Stack

Service

NestJSTypeScriptgRPC

Orchestration

Temporalparent / child workflows

Messaging

Apache PulsarShared subscription

State

MongoDBRedis

Quality

Jestdependency-cruiser

One reward, from campaign to wallet

Every step can be retried or run twice by another pod. Each one is written so a repeat is harmless.

1. TemporalThe parent workflow runs targeting and creates one PENDING delivery row per recipient (idempotent).
2. TemporalA child dispatcher workflow, keyed by campaign and epoch, pages through PENDING rows in chunks of 500.
3. MongoDBEach row is claimed with a compare-and-swap PENDING to IN_PROGRESS. Only the winner continues (mark first, send second).
4. RedisThe budget pool is debited atomically. If the daily cap would be exceeded, the row is held and operations are alerted.
5. PulsarThe winner publishes {deliveryId, epoch}. A rate limiter releases messages gradually so downstream services are not flooded.
6. ConsumerRe-reads the row (the row is the truth, not the message) and rejects anything stamped with an old epoch.
7. MongoDBA second CAS IN_PROGRESS to SENDING. A pod that loses the race acknowledges and skips.
8. ChannelThe reward is delivered and the row is marked SUCCESS or FAILED.
9. ReconcilerA cron pass finds stuck rows after pod crashes, requeues only what is missing, and closes the campaign.

Delivery state machine

Every transition is a conditional update, so two pods can never both move the same row forward.

Key decisions

The core check, simplified

async handle(msg: { deliveryId; epoch }) {
  // the row is the source of truth, not the message
  const row = await deliveries.findById(msg.deliveryId);
  if (row?.status !== 'IN_PROGRESS') return;          // ack and skip

  // epoch fence: messages from an older run never touch the row
  if (msg.epoch < await campaigns.currentEpoch(row.campaignId)) return;

  // CAS: only one pod can move IN_PROGRESS -> SENDING
  const won = await deliveries.updateOne(
    { _id: row._id, status: 'IN_PROGRESS' },
    { $set: { status: 'SENDING' } },
  );
  if (won.modifiedCount !== 1) return;               // lost the race

  const ok = await channel.deliver(row);
  await deliveries.markTerminal(row._id, ok ? 'SUCCESS' : 'FAILED');
}

An open demo with the same architecture (business logic removed): nestjs-temporal-pulsar-demo ↗

Scale

Reliability and operations

• Rate-limited releaseLarge campaigns are drained gradually instead of all at once.
• Single-flight schedulerLocks, claim-based CAS and unique indexes stop a job running twice across pods.
• Never drop silentlyUnknown errors are not acknowledged, so Pulsar redelivers them.
• Layering in CIAn L0 to L3 dependency standard is checked on every change by dependency-cruiser.