From a trader's strategy to a 5,000-customer platform

How 50K bots trade without double orders

For engineers: the services, the order path, how work is split across servers, and the checks that keep real money safe.

Stack

Services

Node.jsExpressPM2 clusterPiscina

Queues

BullMQ256 shards

Market data

WebSocketRedis Pub/Sub

State

MongoDB replica setRedis

Ops

PrometheusGrafana

Exchanges

BinanceOKXBitgetBingX

Five services, one job each

• Socket ServerKeeps live price feeds from 4 exchanges and publishes one price snapshot every 300ms.
• Logic ServerRuns every bot's strategy and risk checks on each 300ms tick, in worker threads.
• DistributionThe admission gate: creates the order ID, blocks duplicates, rate-limits per user and exchange.
• Integration pool3 servers with whitelisted IPs that sign and send orders to the exchange.
• Process OrderApplies order status updates in order, finalises fills and frees the bot for its next trade.

One order, from price tick to fill

Each step is safe to retry. A repeat hits the same order ID and changes nothing.

1. Socket ServerThe Socket Server publishes {SYMBOL: last} for each exchange every 300ms.
2. BullMQA bot job lands on its fixed shard queue. The job ID is botId plus the tick, so the same tick is never queued twice.
3. Logic ServerThe shard owner freezes a price snapshot and runs strategy plus risk in a Piscina worker pool.
4. RedisBefore any side effect, it re-checks that it still owns the shard. If not, the decision is dropped.
5. DistributionCreates the clientOrderId, rejects a second in-flight order for the same bot, and applies the rate limit.
6. Integration poolUpserts the order by clientOrderId first, then sends the signed request to the exchange.
7. Process OrderStatus events are queued by order ID. Old or out-of-order updates are dropped. A final status unlocks the bot.

Key decisions

The ownership check, simplified

async function tick(sid, jobs) {
  const me = `${instId}#${fencing[sid]}`;
  // soft check: don't burn CPU on a shard we may have lost
  if (await redis.get(`logic:shard:owner:${sid}`) !== me) return;

  const prices = snapshot();                       // frozen for this 300ms tick
  const decisions = await pool.run({ jobs, prices }); // strategy + risk, off the event loop

  // hard check: never send if another server took the shard meanwhile
  if (await redis.get(`logic:shard:owner:${sid}`) !== me) return;

  for (const d of decisions) await distribution.post('/orders', d);
}

Scale

Reliability and operations

• Two checks per tickA soft check before computing saves CPU. A hard check before sending stops two owners.
• Back-pressureRising 429s, timeouts or event-loop delay shrink batches and slow the tick automatically.
• Gentle rebalancingShards move only when load is off by more than 10%, one or two at a time, with a cool-down.
• MonitoringEvery service exposes metrics to Prometheus, with Grafana dashboards and alerts.