Short-Message Latency on AI Servers Shows 13x Gap With TCP in Benchmark

A networking protocol called Homa, designed to cut the time short messages spend waiting when multiple AI servers work together, was presented at a developer event. Stanford University’s John Ousterhout shared benchmark results comparing Homa with TCP, the transport protocol in widespread use today, at the AI Engineer event. The core idea is to reduce the latency of the brief exchanges servers send back and forth to one another.

Homa is not a brand new technology. The paper describing it was published in 2018, and research on a module that can be installed on the Linux operating system was released in 2021. This presentation focused on how the protocol can be applied in AI server environments.

Why send short messages first

Ousterhout explained that the communication needs of training an AI model differ from those of serving its answers. In training workloads, what matters is how much bulk data can be pushed through at once. In inference workloads, by contrast, what matters is how quickly the short messages that divide and coordinate work among servers can travel.

Homa treats each request and each response as a single message. The receiving side grants permission for additional transmission of large messages, which regulates how much data is in flight. It also gives priority to messages with less data remaining. Network switches, too, separate queues according to priority. The effect is to reduce the time a short message spends lined up behind a large transfer.

A 13x gap: what exactly was measured

The benchmark Ousterhout presented exchanged messages ranging from very short ones of about 50 bytes up to 1 MB in size. When the times it took for short messages to complete a round trip were sorted from fastest to slowest, the value at the 99th percentile exceeded 1 millisecond (one thousandth of a second) under TCP. Under Homa it came in below 0.1 milliseconds. Ousterhout said the gap in this latency figure is roughly 13x.

Even for the largest messages, Ousterhout said, Homa’s latency was about half that of TCP. What the benchmark measured was the time for a single request and response to travel across the network.

Who can try it today

For now, the people who can actually test it are developers and companies that operate AI servers and networks. The Homa module can be downloaded from a public repository and installed on Linux. Getting full performance, however, requires properly configuring the servers and switches, including their priority queues. The repository notes that an optimization from the 2018 paper, intended for situations where data from many servers converges at once, has not yet been implemented. In September 2026, the detailed mechanism for signaling the start of a transmission to the receiving side also changed.

The next thing ordinary users will want to watch for is whether this gap shows up in the response times of real AI services. Korean companies considering adoption would do well to examine performance in their own network environments alongside installation requirements and the scope of available support.