Olmo-core 3 Lands With 2.7x Throughput for Large-Scale AI Training

The Olmo team released Olmo-core 3, open-source software for training large AI models, on October 1 (local time). The training system has been redesigned around MoE architectures, which compute only the parts of a model that are needed for a given input. The team said its goal is to scale models past the trillion-parameter mark while keeping compute efficient.

Why “compute only part of it” is hard

MoE, short for mixture of experts, routes each input to just a few of many specialized “experts” that divide the work among themselves. That means the whole model doesn’t have to be computed every time. Even so, training still requires the full model to be split across GPU memory and updated. It also requires communication work to send each input to the right expert. The team explained that as models grow, this overhead can eat away at the savings won by reducing computation.

In the team’s tests, increasing total model size roughly tenfold cut training throughput by less than 5%. That came from raising the number of experts from 8 to 128 while keeping the number of experts selected per token (a token is a chunk of text) at 4. Total model size grew from 4.6 billion parameters to 47 billion, but the portion actually used to process a single token stayed nearly the same at about 3.2 billion.

What got faster

Olmo-core 3 changes the training approach so that experts stay put on their GPUs and the needed data is sent to them instead. Previously, model data had to be gathered and then re-split every time a small batch of training data was processed. The team’s earlier model OlmoE was an MoE with 64 experts, while Olmo 3 used most of the model for every input. This overhaul focuses on training larger MoE models.

Image: huggingface.co

In preliminary tests on eight Nvidia B300 GPUs, throughput was about 2.7 times higher than with the team’s previous system. For the 47-billion-parameter model, throughput rose from 19,400 to 52,000 tokens per second per GPU. The system splits experts and model layers across multiple GPUs and bundles small computations so they run in one go.

MXFP8, a format that represents numbers with fewer bits, is also supported. In a test on four B300 GPUs with the workload balanced evenly across experts, training throughput was about 21% higher than with the conventional BF16 approach. Peak memory use fell from 103GiB to 95GiB. The team said it applied MXFP8 only where the payoff was large, and noted that the cost of converting between number formats has to be factored in as well.

Can researchers use it directly?

The team said researchers and developers can use Olmo-core 3 to train their own MoE models and modify it to fit the hardware they have. Larger-scale tests used 512 B300 GPUs and a 1.2-trillion-parameter model. Those tests routed inputs to experts at random, measuring system performance only. A separate short run was also carried out with a 2.38-trillion-parameter configuration.

Image: huggingface.co

The team said it will use an MoE architecture for the next generation of Olmo. The goal is to train on the largest of its own datasets with the longest context length it has used yet. What remains to be seen is whether the speeds from these tests hold up over long, real training runs, and how the resulting models perform in actual use.