OpenAI's COBOL migration benchmark jumps from 5.4% to 86.6%
OpenAI reports stronger coding in COBOL, Fortran and Visual Basic. What MultiLangBench measures, where the gains matter, and why migration still needs review.
On October 19, OpenAI published new evaluations for programming languages with relatively little training material and legacy codebases. On its internal COBOL migration task, GPT-6.1 Sol reportedly reached 86.6 percent accuracy at low reasoning effort, versus 5.4 percent for GPT-5.6 Sol. In MultiLangBench, GPT-6 Astra passed an average of 87.4 percent of tests across 20 languages. These are vendor-run benchmark results, not proof that live banking or enterprise systems can be migrated safely without expert oversight.
Coding assistants have generally performed better on ubiquitous languages such as Python and JavaScript. Older systems in COBOL, Fortran and Visual Basic contain less public training data and often encode poorly documented business behavior.
New build-from-scratch and behavior-preserving migration evaluations measure more extended coding tasks. OpenAI published evaluation materials and internal model results, leaving independent replication and production validation open.
Two evaluations, two different engineering skills
MultiLangBench asks models to build a SQLite-compatible database engine across 20 programming languages rather than just produce plausible snippets. Generated database files are checked against a reference SQLite implementation. OpenAI reports an 87.4 percent average pass rate for GPT-6 Astra, compared with 26.8 percent for GPT-5.6 Sol. Those numbers refer to a particular test environment and reasoning configuration, not to a general claim about shipping reliable applications in any language.
The second benchmark evaluates COBOL-to-C++, Java, Kotlin, Python or Rust migration while preserving observable behavior. That includes subtle business rules such as fixed-point arithmetic and rounding. At low reasoning effort, GPT-6.1 Sol reportedly scored 86.6 percent, versus 5.4 percent for GPT-5.6 Sol; at higher reasoning effort, the newer model reached 97.9 percent. A high score on hidden benchmark tests does not mean a rewritten production system is independently certified or free of regression.
Why COBOL and Fortran still matter
Banking, insurance and government applications often contain decades of accumulated rules that were never fully captured in up-to-date documentation. Generated code can compile and look cleaner while subtly changing transaction amounts, missing-record handling or exception behavior. OpenAI's migration benchmark attempts to expose those errors through output-parity checks. But even carefully designed tests cannot cover every exceptional condition in a large, business-critical legacy system.
Fortran and Visual Basic also remain relevant to scientific and enterprise workflows. In MultiLangBench, OpenAI reports Fortran gains from 21.4 to 83.5 percent and Visual Basic improvements from 27.5 to 89.4 percent between the compared models. This may help teams identify promising pilots, but not rank all competing providers or establish safety. Independent replication, real repositories and regression suites tailored to each organization's processes remain necessary.
A realistic enterprise migration pilot
The right starting point is a bounded subsystem with known interfaces. Capture representative inputs and outputs, numerical edge cases and critical exceptions as a reference. Ask the coding agent to propose a translation, then compare its behavior with the original under deterministic tests. Reviewers who understand the domain must sign off on the preserved semantics. Passing a vendor benchmark cannot replace source review, security checks, staged rollout or parallel operation.
OpenAI says it is releasing the MultiLangBench setup publicly so teams can inspect methodology and language-specific outcomes. For a useful independent evaluation, record exact model versions, reasoning settings, code dependencies and full costs, including human correction. AgentenCode would assess four separate questions: does the code run, does it preserve business outcomes, does it save end-to-end engineering time, and can regressions be audited later? Only that combination supports a procurement or modernization decision.
AgentenCode analysis
The bigger story is that AI coding is improving where skilled maintainers and good documentation are scarce. OpenAI's reported gains are large, but depend on company-run tasks and specific settings. The wrong inference is '97.9 percent means automatic migration is safe.' The practical inference is that narrowly scoped, carefully verified pilots may now be more worthwhile.
Sources and verification
Based on OpenAI's October 19, 2026 primary research post. Benchmark accuracy and cost figures are vendor-reported; AgentenCode has not independently reproduced them or established production equivalence.