OpenAI Says Reads on ChatGPT's Storage Platform Are Now 12x Faster
Median latency for read requests on Habitat, OpenAI's data storage platform, fell from roughly 0.12 seconds to 0.01 seconds. The company credits the gain to keeping frequently requested data closer to users.
On October 23 local time, OpenAI published an account of how it improved read performance on Habitat, its internal data storage platform. According to the company, the median time to serve read requests arriving from around the world dropped from about 0.12 seconds to about 0.01 seconds, a roughly 12x speedup.
Where ChatGPT’s data lives
Habitat is the storage platform that ChatGPT, Codex, and the API, the interface developers use to plug the models into their own services, rely on when they need to fetch data. OpenAI says the platform runs across 40 regions and serves more than 1 billion users. It handles more than 500 petabytes (PB) of data. OpenAI’s largest account on Azure Cosmos DB, the underlying storage service, processes more than 20 million requests per second.
OpenAI also changed how the platform is operated so that multiple workloads share the same storage resources. The company went through a phase of carving out separate storage pools by use case, then returned to shared pools as the default. Heavy-usage items are throttled first, and bulk jobs that write or read large volumes of data at once are given lower priority so that they are rate-limited first when resources run short. OpenAI says that in the months since the change, it has had no incidents caused by capacity allocation or by a single workload monopolizing resources.
How reads got 12x faster
The work focused on data reads, which account for more than 90% of all requests. OpenAI replicated data that is both important and small in size across multiple regions. Frequently requested data was placed in caches, temporary storage kept close to users. The company also says it rewrote the service code in Rust.
The result of those changes, according to OpenAI, was a drop in median read request latency from about 0.12 seconds to 0.01 seconds.
When the stored content changes, the copies held nearby go stale. Habitat collects and distributes data change events that reach millions per second. That stream is used to evict stale cache entries, which are then refilled with fresh data on the next read request. The change events are also used to build indexes that speed up lookups, and to move and analyze data.
Why nearby storage alone isn’t enough
Not all data can be kept close to users. OpenAI points to cloud storage, power, and cooling capacity as the constraints. Only a subset of data can be replicated widely across regions. When a request hits data that isn’t in a nearby cache, it may have to reach all the way out to storage in a distant region.
OpenAI says it is building its own exabyte-scale storage system to hold more conversation data. The design trades away maximum requests per second in exchange for reducing the amount of flash storage required. The question going forward is whether it can hold all of the accumulating conversation data while preserving the read speeds the company has just gained.