Thought Leadership
According to internal testing, the new model translates COBOL programs up to 97.9 percent correctly into modern languages. GPT-5.6 Sol achieved 5.4 percent.
OpenAI has published test results indicating that its new models handle older programming languages significantly better. As examples, the company cites COBOL, Fortran, and Visual Basic. According to OpenAI, COBOL supports more than 40 percent of online banking systems, 80 percent of in-person card payments, and 95 percent of ATM transactions. Fortran is used in applications including weather forecasting and safety analyses for nuclear reactors.
“There is less code and documentation for models to learn from for these languages than for languages like Python and JavaScript, yet they support critical infrastructure and foundational services around the world.”
OpenAI
Database in 20 Languages
For the tests, OpenAI developed the MultiLangBench benchmark. In it, models are tasked with building an SQLite-compatible database engine from scratch across 20 programming languages. GPT-6 Astra achieved an average pass rate of 87.4 percent in testing, compared to 26.8 percent for GPT-5.6 Sol. For COBOL, the figure rose from 10.9 to 72.1 percent, for Fortran from 21.4 to 83.5 percent, for OCaml from 25.6 to 85.8 percent, and for Visual Basic from 27.5 to 89.4 percent.
Migration to Java, Python, and Rust
Additionally, OpenAI tested whether its models could translate COBOL programs into C++, Java, Kotlin, Python, or Rust without altering their behavior. According to company data, GPT-6.1 Sol achieved an accuracy of 86.6 percent at a low reasoning tier, while GPT-5.6 Sol reached 5.4 percent at the same tier. API costs were 61 percent of those for GPT-5.6 Sol. At a higher reasoning tier, GPT-6.1 Sol reached 97.9 percent. According to OpenAI, the benchmark is nearly saturated as a result.
The results stem from internal testing by OpenAI; independent evaluations are not yet available. According to BleepingComputer, migrations in production environments continue to require review by specialists and extensive testing. According to a Harvard study, human review of generated code is increasingly becoming the bottleneck in AI-assisted software development.
(Editorial Team)
