Ai2 Gave Projects GPU Time Budgets and Cut Hands-On Repairs by 74%

Ai2, the AI research institute, has described a new way of divvying up its research GPUs: instead of ranking jobs by priority, it allocates “time budgets.” Each project gets a share of GPU hours, and the order in which jobs run is adjusted based on how much of that share has actually been used. Even long-running jobs can be asked to yield their spot once they have run for a certain amount of time.

Ai2 operates several thousand Nvidia H100, B200 and B300 GPUs. Individual clusters range from 88 to 1,024 GPUs. Roughly 150 in-house researchers use them for work such as large-scale AI training and robotics research. According to Ai2, the GPUs requested by queued jobs add up to two to three times the capacity available.

When 100% of jobs are “top priority”

Under the old system, research teams set job priorities themselves and could flag jobs that must not be interrupted. The result, Ai2 says, was that every job actually running ended up marked “high priority.” Some researchers even parked empty jobs that did no work at all, simply to hold GPUs in reserve for the moment they needed them. Unable to launch a debugging experiment on demand, they claimed a seat in advance.

Dedicating GPUs to important projects had its own limits. While that team prepared its experiments, the GPUs sat idle and other teams still had to wait. Long-running jobs also blocked hardware maintenance. Ai2 says its on-call engineers spent most of their support time negotiating with researchers to wrap up jobs on machines that needed servicing.

What changes when you split time instead

In the new system, administrators divide GPU time budgets to match the shape of the research organization and its projects. By default, the job scheduler looks at usage over the past seven days. Jobs from teams that have used less than their allocated share run first; teams that have used more go to the back. Ai2 says it designed the system so that each team can receive its allocated share on a weekly basis, provided it submits jobs steadily enough to fill that time.

Image: huggingface.co

From a researcher’s point of view, the key difference is no longer being capped purely by how many GPUs you can use at once. If you have used little for a while, you can be allocated more later when your workload piles up. Conversely, running an empty job just to hold a seat eats into your own team’s budget. Spare GPUs can also be used without drawing down a budget, though jobs run that way can be stopped at any time as soon as budget-backed work arrives.

When do long training runs get stopped?

When submitting a job, researchers specify the minimum run time needed to make meaningful progress. Budget-backed jobs are not interrupted during that window. After that, depending on other teams’ usage and budgets, a job either keeps running or gives up its spot. Jobs that can be resumed go back into the queue automatically. Ai2 says this made it possible to drain jobs off machines needing maintenance one by one, cutting the number of repairs requiring human intervention by 74%.

Image: huggingface.co

The shift is from a race to grab scarce GPUs first to deciding in advance how much each line of research gets to use. Ai2 says it moved the resource allocation conversation out of per-job negotiations and into budget decisions made by administrators. The thing to watch next is how wait times change from project to project. Ai2 itself notes that giving one researcher more time means less for another, and that any policy change can invite a fresh round of workarounds.