Head-of-line blocking in the control plane

Tensorlake recently pushed sandbox scheduling throughput up 12x by replacing a single replicated state machine with three independent Raft lanes. The number is striking, but the mechanism is the interesting part: they found that their scheduler was blocking itself, and the fix was the same decomposition that distributed databases applied to storage a decade earlier.

The single-pipeline problem

A replicated state machine (RSM) gives you strong consistency cheaply: every participating node sees the same log entries in the same order, so the state machine they apply is identical everywhere. That ordering guarantee is also the core constraint. Entry n cannot be committed until the majority has acknowledged entry n-1. Nothing in the pipeline skips the queue.

Tensorlake's scheduler had three fundamentally different kinds of work flowing through that one queue:

1. User intent: create, delete, or update a sandbox. Heavy, potentially bursty, generated by external API traffic.
2. Placement decisions: which dataplane host takes which sandbox. Generated by the scheduler itself in response to intent, roughly proportional to intent volume.
3. Dataplane heartbeats: status updates from hosts confirming that sandboxes are running. Constant, lightweight, time-critical.

Under normal load this is fine. Under a burst of 100,000 concurrent creation requests it is not. The creation intents flood the log. The log commits them in order. Placement decisions pile up behind them. Heartbeats pile up behind placement decisions. The scheduler, waiting for heartbeats that are stuck in the queue, starts treating healthy hosts as unresponsive. Scheduling decisions degrade. The system is not overloaded; it is blocked on itself.

The pipeline did not fill up because it lacked capacity. It blocked because messages that needed to be independent were forced to be ordered.

Multi-Raft: the decomposition

The fix is to stop pretending that ordering between lanes is required. User intent has no causal relationship to an unrelated host's heartbeat. Placement decisions for sandbox A have no ordering dependency with status updates for sandbox B. The global ordering the single RSM enforced was always unnecessary across these classes; it was only necessary within each class.

Multi-Raft makes that explicit by giving each lane its own independent Raft consensus group with its own log, its own leader election, and its own commit sequence. The three groups run concurrently on the same physical scheduler nodes.

A burst flooding lane 1 advances its own log index independently of lanes 2 and 3. Lane 3 heartbeats commit without waiting for any entry in lane 1, regardless of how deep the lane 1 backlog runs.

The same idea in storage engines

TiKV and CockroachDB arrived at Multi-Raft from a different direction. A single Raft group managing the entire database keyspace puts one node in charge of all writes, everywhere. That leader saturates and becomes the global bottleneck. The solution was to slice the keyspace into small ranges, roughly 64 to 96 MB each, and give every range its own independent Raft group. Reads and writes to non-overlapping ranges proceed in parallel across the cluster with no coordination between them.

The operational challenges that follow are real. Tens of thousands of independent Raft groups on a single machine would normally mean tens of thousands of independent heartbeat timers, one per group, broadcasting to all followers. On a cluster with many overlapping group memberships between the same pairs of nodes, this is a network storm.

Both TiKV and CockroachDB address it with coalesced heartbeats: the transport layer batches all Raft messages between a given pair of nodes, including heartbeats from every group they share, into a single network packet. The logical independence of the groups is preserved; the physical transport is shared.

Storage writes face a similar tension. Each Raft group needs to persist its log entries durably before acknowledging a commit. Writing to thousands of separate log files in parallel would fragment the disk I/O into random seeks.

Both engines solve this by routing all Raft log appends, regardless of which group they belong to, through a single underlying key-value store, with the group ID as a key prefix. Multiple concurrent log entries from different groups land in a single WriteBatch flush, preserving sequential disk throughput while maintaining logical separation above it.

Tensorlake is running the same playbook at the control plane layer rather than the storage layer. Three groups instead of thousands, which sidesteps most of the heartbeat and storage batching complexity, but the core architectural move is identical: identify which ordering constraints are load-bearing and which are accidental, then remove the accidental ones.

The same decomposition in query execution

A closely related insight shows up in how DuckDB handles graph queries. Given a problem that looks like it requires recursive graph traversal, a linear sequence of events aggregated by window, DuckDB sidesteps the graph entirely and uses a SQL window function instead: a single sorted pass over flat columnar data, never touching the graph topology.

The Cypher-native approach (LadybugDB's Kleene star operator, matching variable-length paths through the event chain) resolves around 6.6 million adjacency steps on a single thread and takes close to three minutes on 15 seasons of MLB play-by-play data. The window function takes three seconds.

The structural move is the same one. Graph traversal imposes an ordering on the computation because each step depends on the previous hop. A window function over a partitioned, sorted relation removes that dependency because the data model makes the structure explicit: there is one partition per half-inning, one sort key per play sequence number, and the aggregation is a single linear pass. What looks like a graph problem is actually a sequence problem, and sequence problems do not need the graph machinery.

In all three cases, the performance gain came from removing an ordering constraint that the problem did not actually require. The constraint was in the implementation, not in the problem.

When decomposition is valid

Multi-Raft is correct here because the independence claim holds: user intent for sandbox A carries no ordering relationship to a heartbeat from sandbox B's host. If the independence claim were false, if heartbeats had to be committed after all creation intents that might affect their validity, splitting the lanes would break correctness, not just performance.

Diptanu Choudhury, one of the Tensorlake engineers, put it directly: "the intent of creating sandboxes from users and dataplanes reporting the control plane is mostly independent." That word "mostly" is doing real work.

The scheduler can receive a heartbeat confirming that sandbox A is running before the log entry creating sandbox A is committed in lane 1, because the placement decision in lane 2 and the host acknowledgment in lane 3 follow from physical events, not from log ordering. The system reconciles the lanes through the scheduler's own state machine logic, not through the Raft ordering guarantee.

This is the engineering judgment that matters most: not "can we decompose this" but "at what grain does the ordering guarantee become load-bearing." Get that wrong and you trade a performance problem for a correctness problem. Get it right and you remove a constraint that was never needed.

When the complexity is worth it

A single Raft group on modern hardware can sustain roughly 10,000 to 50,000 small writes per second before the leader becomes the bottleneck, depending on payload size, network round-trip, and disk.

For a control plane handling a few hundred operations per second with homogeneous traffic, splitting into multiple groups adds operational overhead (independent leader elections, split-brain surface across groups, more state to reconcile) for no measurable gain. The single RSM is the right default.

The signal that you have Tensorlake's problem specifically is not throughput saturation. It is latency divergence under burst: your time-sensitive operations (heartbeats, health checks, lease renewals) show p99 latencies that spike in direct correlation with bulk operation volume, while your hardware metrics show spare capacity. The queue is not full; it is ordered wrong.

Adding nodes or increasing Raft log batch sizes will not help. The bottleneck is structural, and more resources applied to the same structure produce the same blocking at a higher ceiling.

The numbers that made the decomposition worthwhile for Tensorlake: burst volume an order of magnitude above steady state (100,000 concurrent creates during a load test), a heartbeat timeout threshold in the low hundreds of milliseconds, and a single RSM commit latency under burst that exceeded that threshold. All three conditions together. Any one of them alone is manageable with tuning. The combination is not.

TiKV and CockroachDB are at a categorically different scale: petabyte keyspaces, tens of thousands of ranges, Raft groups numbered in the hundreds of thousands per large cluster. The coalesced heartbeat and WriteBatch machinery they built exists because at that group count the naive implementation generates a network storm and fragments disk I/O.

Tensorlake's three-group design sidesteps all of that. The core insight is shared; the operational cost is not remotely comparable.

The result

Twelve times the scheduling throughput. The number reflects what happens when an accidental serialization point is removed from a system that otherwise had the capacity to handle the load. The work was not in adding resources; it was in identifying the constraint and proving it unnecessary.