- A security researcher has discovered a novel security flaw in the Linux kernel using the OpenAI o3 reasoning model.
- The new vulnerability has been documented under CVE-2025-37899. An official patch has also been released.
- o3 processed 12,000 lines of code to analyze all the SMB command handlers to find the novel bug.
A security researcher named Sean Heelan has found a new zero-day vulnerability in the Linux kernel by using OpenAI’s powerful o3 reasoning model. This is the first time an AI model has discovered a security flaw in a complex software system like the Linux kernel which runs on millions of servers and computers. In fact, the vulnerability has been documented under CVE-2025-37899 .
Heelan writes in a blog post that he was auditing the ksmbd module for vulnerabilities using the OpenAI o3 AI model through the API without any tool use. ksmbd is “ a linux kernel server which implements SMB3 protocol in kernel space for sharing files over network. “
In this case, o3 understood concurrent connections to the server and found “ a location where a particular object that is not referenced counted is freed while still being accessible by another thread. ” Basically, o3 identified a critical “use-after-free” vulnerability in the handler for the SMB ‘logoff’ command.
o3 processed all SMB command handlers, which are about 12,000 lines of code, consuming around 100K tokens. A patch to the Linux kernel has already been committed and merged into the official Linux kernel repository on GitHub. This is the first instance where an AI discovers a bug, a human verifies it, an official patch is released, and the vulnerability is closed.
Interestingly, the researcher found the novel security bug while evaluating AI models like Claude 3.7 Sonnet , Claude 3.5 Sonnet, and OpenAI o3 on another security flaw — Kerberos authentication vulnerability (CVE-2025-37778). Heelan writes that o3 found the Kerberos vulnerability in 8 of the 100 runs; Claude 3.7 Sonnet found it 3 out of 100 runs, and Claude 3.5 Sonnet couldn’t find it in 100 runs.
Lastly, the researcher cautions that “o3 is not infallible,” but recent reasoning AI models have made a significant leap in understanding large codebases. If you have a project below 10K lines of code, models like o3 can help you solve problems. And for vulnerability research, new reasoning models can make you “ significantly more efficient and effective. “
Passionate about Windows, ChromeOS, Android, security and privacy issues. Have a penchant to solve everyday computing problems.
Add new comment
Name
Email ID
Δ
