pyRPC
← Back to Blog

Batched RPC requests, end to end

·8 min read

A dashboard that renders five widgets issues five RPC calls. Five HTTP round trips, five connection setups, five chances for one slow request to dominate the waterfall. v0.13.0 ships batching as a transport optimization, and the phrase is chosen carefully, because the interesting decisions are all about what batching is not.

Timeline diagram: three operations issued in one tick coalesce into a single POST array and resolve independently

Client: collect the window, flush it

httpBatchLink exploits a JavaScript scheduling fact: synchronous code that issues several calls runs inside one event-loop turn. Operations arriving in that window are queued; the first enqueue schedules a setTimeout(0) flush:

return new Promise<OperationResult>((resolve, reject) => {
 current.pending.push({ operation, resolve, reject });

 if (current.pending.length >= current.maxItems) {
 void flush(current); // size cap reached
 return;
 }
 if (current.timer === null) {
 current.timer = setTimeout(() => void flush(current), 0);
 }
});

The result: calls in the same tick share one POST carrying a JSON array; each caller keeps its own promise. A missing slot in the response array rejects that operation with a clear error instead of hanging.

Server: dispatch sequentially, respond in order

The interpreter accepts either shape. A dict dispatches exactly as before; a list iterates through the normal single-operation path and collects responses positionally:

MAX_BATCH_SIZE = 100

if isinstance(payload, list):
 if len(payload) > MAX_BATCH_SIZE:
 return [invalid_request(
 f"Batch too large: {len(payload)} operations (max {MAX_BATCH_SIZE})"
 )]
 return [await handle_single(op, router) for op in payload]
  • In order, so index mapping stays trivial on both sides.
  • Capped at 100, so an accidental Promise.all over ten thousand rows cannot become one giant request.
  • Per-operation errors: one failing procedure returns its error object in its slot; siblings are untouched.

What batching deliberately is not

The docstring on httpBatchLink spends more words on semantics than mechanics, and that ratio is the design:

  • Not a transaction. Nothing rolls back when operation three fails. Batching shares a socket, not a database scope.
  • Not parallel execution. The server dispatches sequentially. Predictable ordering beats speculative concurrency for state-mutating procedures; if you need isolation or parallelism, issue independent requests.
  • Not a new procedure kind. No special casing downstream, auth, validation, and introspection see exactly what they saw before, per operation.

The payoff you actually notice

With TanStack Query adapters, a page mounting four queries fires them in one tick, which now means one request. Latency drops from N×RTT toward 1×RTT without any API change: no batch() wrapper, no explicit grouping. You write ordinary calls; the transport notices they are neighbors.

That is the test for transport features: invisible when unneeded, automatic when applicable, honest about limits (the size cap exists server-side precisely because client defaults can be wrong). Batching clears all three bars.