Why I Built a Message Broker on QUIC Instead of TCP

Head-of-line blocking is one of those problems that's easy to describe and annoying to actually fix, because the fix usually means replacing a transport layer that everything else was built assuming would stay put.

The shape of the problem

TCP guarantees ordered, reliable delivery over a single byte stream. That guarantee is exactly what makes it usable for almost everything — and exactly what causes head-of-line blocking. If packet 4 in a stream is lost, packets 5 through 20 that already arrived have to sit in the kernel's receive buffer, unusable, until packet 4 is retransmitted and arrives. It doesn't matter that packets 5 through 20 belong to a completely unrelated logical message; TCP doesn't know about "messages," only bytes in order.

Kafka and NATS both build their fan-out and multiplexing on top of TCP connections. When a broker multiplexes many topics or subscriptions over one connection — which is the common, efficient thing to do — a single retransmit stalls delivery for every subscription sharing that connection, not just the one whose packet was lost. Under normal network conditions this rarely matters. Under real-world loss, especially cross-region or cross-cloud where loss rates are non-trivial, it becomes a tail-latency problem that's very hard to diagnose because it looks like "sometimes everything on this connection just pauses for a bit."

The usual workarounds are connection-level: open more TCP connections per consumer group, shard subscriptions across connections, accept the operational complexity of managing many more sockets. It works, but it's treating a transport-layer problem with an application-layer patch.

What QUIC actually changes

QUIC runs over UDP and implements reliability and congestion control itself, but critically, it does so per stream, not per connection. A QUIC connection can carry many independent streams, and loss on one stream only blocks that stream — every other stream on the same connection keeps delivering data as it arrives. This is the same property that made QUIC attractive for HTTP/3 (a lost packet for one page resource shouldn't stall every other resource loading over the same connection), and it maps almost directly onto the messaging use case: give each subscription its own QUIC stream, and a slow or lossy subscriber stops being everyone else's problem.

That's the core idea behind QuicMQ, a message broker I built in Rust specifically to test this out in practice rather than in theory. Each subscription gets its own QUIC stream over a shared connection per client, so:

• A consumer that's slow to ACK, or a network path with elevated loss to one particular consumer, no longer stalls delivery to every other subscription multiplexed over the same connection.
• New streams can open without the handshake cost of a new TCP+TLS connection, since QUIC's connection-level handshake is amortized across all streams.
• Connection migration — a QUIC feature that lets a connection survive a client's IP or network change — turns out to matter more than expected for consumers running on infrastructure that can shift networks mid-session.

What it took to make durable

Independent streams solve head-of-line blocking, but a broker still needs the guarantees people actually rely on Kafka for: durability and replay. QuicMQ writes every message to a write-ahead log before acknowledging it, so a broker restart doesn't lose in-flight messages, and consumers can replay from an offset the same way they would against Kafka. It also exposes Prometheus metrics per stream — which, once you have per-subscription isolation at the transport level, becomes a genuinely more useful signal than per-connection metrics ever were, because you can finally see which specific subscription is degraded instead of inferring it from a connection-wide slowdown.

Where this does and doesn't help

This isn't a claim that QUIC-based brokers should replace Kafka for high-throughput, single-consumer-group log processing, where Kafka's design center is genuinely a better fit. It matters most for fan-out topologies with many independent, heterogeneous consumers — the exact shape you get in multi-tenant or multi-region systems where you don't control every consumer's network quality, and one bad connection shouldn't degrade the rest.