As AI Finds Hidden Software Bugs, Defenders Turn to Mathematical Proof

Imagine opening a photograph in an application that has successfully displayed thousands of images. This file contains an unusual combination of dimensions and compressed data. The code reading it calculates a storage size incorrectly, then reaches beyond the memory set aside for the image. The result could be a crash, exposed information or, under the right conditions, an opportunity for an attacker to take control of part of the program. A familiar task has encountered an unfamiliar path through its code.

Software developers have spent decades hunting such paths. Artificial intelligence (AI) is making some of that search easier to automate, including in mature software that has already endured years of scrutiny. It is also entering the harder defensive business of proving that specified failures cannot occur. An October 22 report from Anthropic and Theorem brings those two developments together: machines that search for weaknesses can also do some of the mathematical work needed to close them. [1] [7]

The same faulty image reader can pose different risks depending on where it runs and who can supply its files. A bug is a mistake in how software behaves. A security vulnerability is a weakness that can undermine access controls, expose information, corrupt data or interrupt a service. Turning it into an exploit requires a usable way to trigger that weakness and cause an unauthorized result. Whether an attacker can reach the faulty code, control the relevant input and overcome other defenses determines the danger. [5] [8]

Memory handling is one source of trouble. Programs reserve regions of memory for particular objects and track how large those regions are and when they can be reused. Reading or writing outside an allocated region is an out-of-bounds access. Another class of defect is a missing permission check: a service might correctly check ownership when someone opens a document, yet omit the same check on its export path. Each weakness breaks a different rule, even if both begin with an apparently ordinary request. [5]

Sequence matters, too. Consider a hypothetical application that cancels a request and frees its working memory, then receives a delayed completion message that still refers to that memory. Each action can look reasonable on its own; their order creates the fault. Modern applications pass work among interfaces, parsers that interpret input, shared libraries and operating-system services. A component may assume that another has already validated a value or preserved an object. Errors accumulate at those handoffs.

These defects can survive for years because testing must explore combinations: file contents, sizes, permissions, earlier actions, timing and dependency versions. Most customers follow common paths. A faulty branch may require an improbable sequence that ordinary use rarely exercises. Developers can also misunderstand the intended rule itself. A test suite faithfully checking an incomplete design may approve behavior that should have been forbidden.

The existing defenses attack this problem from different directions. Code review asks people to inspect the logic and its assumptions. Tests execute chosen cases and compare the results with expectations. Fuzzing automates a broader search by repeatedly generating or changing inputs to provoke failures. Google's OSS-Fuzz combines fuzzing with sanitizers, tools that add checks for errors such as invalid memory access. Each technique produces evidence about behavior; the quality of the questions and the code exercised still shapes what it finds. [6]

How AI changes the search

AI changes how much directed investigation a team can attempt. An agent can read related code, form a hypothesis about a broken assumption, generate a focused test, run it and revise the hypothesis. Multiple jobs can examine different components in parallel. This combines search with reasoning about why one component's behavior might violate another's expectations. Engineers still need a working testing system, triage and fixes for the findings to become protection. [7]

Mozilla's engineers described that process in May. They said Firefox 150 included fixes for 271 bugs identified by Claude Mythos Preview. Their examples included defects roughly 15 and 20 years old, as well as problems requiring reasoning across browser processes. Mozilla also described building reproducible tests and filtering findings through its security workflow. Some reported sandbox escapes began with an assumed compromise of the restricted content process. Reaching the more privileged process was a further step along an attack path. [7]

Following a suspected flaw toward a working attack involves several stages. Discovery identifies a weakness. Reachability establishes that relevant inputs can get to it. Reproduction demonstrates the faulty behavior. Exploitability asks whether that behavior gives an attacker a useful capability, such as reading protected data. Exploit chaining connects capabilities to cross additional barriers. Actual compromise occurs when an attacker achieves unauthorized control or access in the target system. A crash report answers an earlier question than a demonstrated takeover. [8]

The independent ExploitBench project tests how far AI agents can take a known vulnerability. Its 41 patched cases concern V8, the engine that executes JavaScript in Chrome and other software. Agents receive the vulnerable build, the fixed build and the repair patch, then try to develop an exploit. The authors currently report that Mythos Preview and GPT-5.5 have achieved arbitrary code execution in this controlled setting with V8's internal security sandbox enabled. The experiment measures work on supplied, known bugs in V8's standalone environment; complete browser or device compromise involves further boundaries. [8]

Reading unfamiliar code and constructing an unusual test have traditionally consumed scarce specialist time. AI can shoulder portions of that labor and repeat the search across more targets. Attackers benefit from the same capability where they can obtain and use it. The advantage remains uneven across models, software and operating conditions, but obscurity becomes a weaker reason to expect a reachable flaw to stay hidden.

What a proof adds

Formal verification uses mathematical reasoning to establish that a program satisfies a stated rule across every case covered by a model. For the hypothetical image reader, tests could check thousands of image lengths. A proof could establish that every accepted length keeps each read within the allocated memory, under precisely stated assumptions. Testing examines cases; a proof establishes the rule governing the covered cases.

A memory-safety rule addresses where a program reads or writes. The missing export permission check needs a different rule: every export must require the requester's authorization. A perfectly checked memory theorem can coexist with a design that grants the wrong person access. Engineers must decide which failures matter and express those requirements accurately. The proof then gives unusually strong evidence about those requirements within its defined world.

Anthropic and Theorem report applying this approach to 239 testing entry points across 123 open-source libraries, patching discovered bugs and producing proofs for the resulting programs. An entry point is a particular interface through which a test invokes library code. It might exercise a parser or a decompression routine. The reported count describes selected paths into these libraries; the rest of each library has its own coverage questions. [1]

The work starts with existing fuzz-testing harnesses, the small programs that feed inputs to a library and check what happens. The technical appendix describes guarantees against faults and aborts, including failed assertions, which are explicit checks that a rule holds. It also relates ordinary compiled programs to versions containing AddressSanitizer, which checks memory handling. Observable behavior includes input and output, exit behavior and final file contents. This carries assurance beyond a model of source code to the particular executable instructions being assessed. [2]

Claude writes the proof; Rocq, a separate proof assistant, checks its derivation against the mathematical statement. A persuasive explanation from an AI carries little weight in that process. The checker must accept the proof according to its rules. The released repository identifies binaries, statements and toolchain versions so others can inspect the claim and reproduce the checking procedure. When inspected for this article, it listed a public subset of 61 entry points across 26 libraries. The artifacts were examined; their proofs were not independently rerun. [3]

The reported campaign concerns single-threaded programs, which execute one stream of instructions, statically linked so their library code is incorporated before launch. Their binaries contain RISC-V processor instructions, the basic operations the hardware executes. The research uses a model of Linux process behavior and a fixed memory budget. The appendix sets an input limit of less than 64 mebibytes, about 67 million bytes. Those choices make a tractable proving problem. Concurrency, dynamically loaded code and other instruction sets introduce further obligations. [2] [4]

Some released proofs establish agreement between ordinary and sanitizer builds under specified memory-budget conditions; halting statements establish allowed termination separately. An engineer using a proof must read the actual statement attached to the released binary. That is how a broad promise of safer software becomes a concrete guarantee about a particular build, set of inputs and failure class. [3]

Underneath the proof sits a trusted foundation. The statement must describe the desired property; the processor and Linux models must correspond to real execution; and the checking software and the system running it must behave correctly. The public Linux model documents its supported behavior and known gaps. Independent scrutiny of those definitions therefore matters alongside checking the generated derivation. A mistaken assumption in a shared model could affect many otherwise valid proofs. [4]

An attacker searches for one useful route through the program's behavior. A proof can remove an entire covered class of routes from that search by establishing that the forbidden event never occurs within its scope. Defenders still need separate assurances for other properties and interfaces. The value comes from knowing exactly which ground has been secured and which ground still needs investigation.

The cost of keeping a guarantee

Anthropic says producing a proof can consume hundreds of millions of tokens, the chunks of text models process and generate. In a separate 36-task experiment, each task received three attempts with a five-million-token budget per attempt. The company reports that Opus 5.5 reached half the tasks with about 40 percent fewer tokens than Opus 5. A separate proof-sharing experiment used 17 percent fewer tokens. [1]

Those smaller tasks used inputs limited to 64 bytes and a simplified sanitizer, according to the appendix. Their efficiency results therefore describe a reduced proving workload. They offer evidence that proof production can improve, while the full campaign still needs a complete account of attempted targets, failed runs, human effort and dollar costs before a buyer can calculate a production return. [2]

The useful measure is the cost of an accepted guarantee that matters to users and survives maintenance. That includes failed AI attempts, checking compute, the time spent choosing specifications and independent review of patches. A change that makes a theorem easier to prove can also change behavior customers need. Once a compiler, dependency or application changes, the team must establish which guarantees still apply and which require new proof work.

For a service that accepts uploaded files, the practical choice could begin with its exposed image parser. Engineers would identify the exact decoding interface and build, define a memory-safety property, review any repairs and check the resulting proof. Access-control review would address who may retrieve the files. Each release would carry its coverage record forward. As AI expands the search for obscure weaknesses, that maintained record would give the operator a concrete answer about one important attack surface: the property established, the code covered and the assumptions still being relied upon.

Sources

1. Anthropic formal verification research report

2. Technical appendix to the research report

3. Public OSS Verify proof repository

4. Theorem Linux process semantics

5. MITRE 2025 Common Weakness Enumeration Top 25 archive

6. Google OSS-Fuzz documentation

7. Mozilla engineers on hardening Firefox with AI

8. ExploitBench authors' methodology and current results