OpenAI Boosts Coding on Legacy Languages Like COBOL, Hitting 87.4% Pass Rate
GPT-6 Astra averaged an 87.4% pass rate on a benchmark that asks models to build a database engine in 20 different languages. OpenAI also says it improved accuracy and cut costs on a test that ports COBOL programs to other languages.
OpenAI announced on October 19 (local time) that its models have gotten better at writing code in programming languages with scarce training material, such as COBOL, Fortran and Visual Basic. The company released two benchmarks alongside the announcement: one measures the ability to build new programs, the other the ability to port existing programs to a different language.
Building in 20 languages: how many tests passed?
The new benchmark, MultiLangBench, asks models to build a database engine from scratch in 20 languages. The program has to store and retrieve data while remaining compatible with SQLite, the widely used database. OpenAI used the benchmark to gauge coding ability language by language.
According to OpenAI, GPT-6 Astra recorded an average test pass rate of 87.4% when set to its maximum reasoning effort. That is 60.6 percentage points higher than the 26.8% posted by GPT-5.6 Sol. Astra cleared an 83% pass rate in 19 of the 20 languages. In COBOL, the hardest of the group, it scored 72.1%. The company said it is publishing the benchmark’s structure, its tests and the per-language results on GitHub.
On the same benchmark, the company said GPT-6 Luna delivered a higher pass rate than GPT-5.6 Sol while costing less. Luna’s average pass rate at maximum effort was 40.3%. The estimated average cost of solving a single benchmark task was $0.77, roughly 78% less than the $3.48 estimated for GPT-5.6 Sol. Those figures reflect the cost of running the benchmark through the API, the method developers use to plug a model into their own services.
How accurate is porting COBOL to another language?
The second benchmark covers converting existing COBOL programs into other languages. After the port, the program must behave exactly as the original did. The target languages are C++, Java, Kotlin, Python and Rust. The test measures not just the ability to write fresh code but whether existing functionality survives the move intact.
OpenAI said GPT-6.1 Sol scored 86.6% accuracy even at a low effort setting, compared with 5.4% for GPT-5.6 Sol at the same setting. GPT-6.1 Sol’s API cost was 39% lower than the comparison model’s. When configured to spend more effort on each task, accuracy climbed as high as 97.9%, the company said.
What to watch when putting this to work
Developers can call these models through the API and hand them coding tasks in these languages. The key question is whether a ported program faithfully preserves the original behavior. OpenAI explicitly stated that migrating software already running in production requires expert review. It also stressed the need for testing that covers business rules and the other programs connected to them.
The core of the announcement is that AI has gotten better at both writing and porting programs even in older languages with little training material to learn from.