From target reproduction to open-ended discovery

Evaluation of LLM bug-discovery capability has evolved through four generations. The first two operate on static code: binary classification assigns a vulnerability label, while structured reporting adds the expected class, location and description. Neither executes the model’s result.

The third generation introduces dynamic execution. A model generates an input for a sanitizer-instrumented harness, but success is typically defined against one predefined vulnerability. Valid crashes elsewhere may be discarded, grading may depend on a developer-provided patch, and faults originating in the harness may be mistaken for findings in the project under test.

FuzzingBrain-Bench represents a fourth generation: open-ended discovery. It credits each distinct crash signature that satisfies the benchmark’s validity and reproducibility criteria, including crashes beyond the vulnerability originally associated with a challenge.

Design principles

Challenge construction

Each challenge begins with a published finding reached through a fuzzing harness. Findings are drawn from FuzzingBrain V2, QuartetFuzz, OSS-Fuzz, and project issue trackers, pull requests and security advisories. Four artifacts are extracted: source at the vulnerable commit, harness code, sanitizer configuration and a reference proof-of-concept (PoC).

The harness is compiled under the selected sanitizer and packaged with the source into a Docker image. The reference PoC is executed three times inside the image; the challenge is retained only when all three executions produce the same crash signature.

Before publication, the reference PoC, report, git history and identifying build commands are removed. The published image contains the project source, harness source, benchmark configuration and sanitizer-instrumented harness binary.

Evaluation environment

The agent loop runs on the host, communicates with the model API and enforces the turn and time budgets. The challenge runs in a container connected to the host through MCP over standard input and output.

The harness binary and grading configuration reside in an inaccessible oracle directory. Candidate inputs can reach the binary only through run_poc_on_harness. This restriction prevents direct fuzzing of the graded binary from dominating the evaluation.

Each episode is limited to 100 turns and 1,800 seconds. It continues after the first valid crash so that the model can search for additional signatures.

Crash validity

A candidate execution is classified as a crash when it terminates in one of the abnormal conditions recognized by the benchmark. A crash is valid unless its fault site lies in the harness itself or both output streams are empty.

Crash signatures and deduplication

Because multiple inputs may exercise the same failure, FB-Bench represents each valid crash by a signature and deduplicates signatures within a challenge run. A signature consists of a normalized fault class followed by up to three relevant function names.

The signature is designed for stable deduplication rather than root-cause identification. Two signatures may still correspond to one underlying defect when it is reached through different paths; the benchmark therefore reports distinct crash observations, not confirmed unique vulnerabilities.

Reproducibility and novelty

Corpus composition

V1 contains 77 challenges from 43 open-source projects, each constructed from a commit with a confirmed vulnerability. The corpus includes 36 C, 32 C++ and 9 Java/JVM challenges.

Three languages

The C and C++ challenges contain low-level memory faults, while the Java/JVM subset contributes exception and resource-exhaustion cases.

Five fault detectors

Each harness is built with the detector associated with its source finding. Sixty-eight challenges use a libFuzzer engine and nine use Jazzer.

Fourteen bug classes

45 challenges are memory-safety defects; the other 32 are denial-of-service and other faults, including uncaught JVM exceptions, reachable assertions and undefined behaviour.

Difficulty and scoring

Challenges that all reference models crash should not receive the same weight as challenges that none of them crash. Each challenge therefore receives a difficulty coefficient D from 1 to 5, derived once from a fixed panel consisting of Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8. Tier assignment uses uncapped signature counts.

D4 and D5 contain 24 challenges and account for 327 of the 579 available points. Consequently, a signature on a D5 challenge contributes five times as much as a signature on a D1 challenge.

Evaluation results

We evaluated Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8 on all 77 challenges. Each model completed one blind episode per challenge with a maximum of 100 turns and 1,800 seconds; every candidate input was executed three times.

Opus obtains the highest score, 196 of 579, and produces a valid crash on 60 of 77 challenges. Sonnet scores 156 and crashes 50 challenges; Haiku scores 58 and crashes 35. None of the three models crashes 13 challenges.

Performance by difficulty tier

All three models crash every D1 challenge, while none crashes a D5 challenge. Differences emerge in D3 and D4: Opus crashes 9 of 11 challenges in each tier, compared with 6 and 2 for Sonnet and 2 and 0 for Haiku.

Sonnet scores more points than Opus in D1 and D2, whereas Opus scores 96 points in D3 and D4 combined, compared with 42 for Sonnet. The 13 D5 challenges account for 195 points, none of which is obtained by the evaluated models.

Turn-budget utilization

Opus uses a median of 51 of 100 turns, whereas the median for Sonnet and Haiku is 100. This difference may partly explain why Sonnet obtains more points in D1 and D2 despite a lower total score.

Effect of the per-challenge cap

Removing the cap increases the scores of Opus, Sonnet and Haiku by 33.2%, 30.8% and 12.1%, respectively, without changing their ranking. The cap limits the contribution of challenges that yield many signatures.

Cost, token usage and runtime

Opus’s mean cost increases from $1.10 per D1 challenge to $4.54 per D5 challenge, reflecting earlier termination on challenges where it finds crashes sooner. Sonnet and Haiku show less variation across tiers because they use most of the turn budget on nearly every challenge.

Input tokens exceed output tokens by approximately 84:1 for Opus, 143:1 for Sonnet and 188:1 for Haiku. Opus’s token use and duration increase with difficulty. Sonnet’s output falls from 56.2k tokens in D1 to 28.1k in D5 as its interaction shifts from constructing inputs to shorter reconnaissance commands.

Run it yourself

Each challenge is a self-contained public image that performs grading offline. Running the benchmark requires Docker, Python 3.10 or newer, and an API key for the model under evaluation.

git clone https://github.com/fuzzingbrain/FuzzingBrain-Bench
cd FuzzingBrain-Bench && pip install -e .

echo "ANTHROPIC_API_KEY=sk-ant-..." >> .env
fb-bench run json-java-02 --model claude-haiku-4-5
# OpenAI, Google and DeepSeek models work the same way, routed by name

Agent backends use the same MCP server, and the full corpus can be evaluated with resumable sweeps:

fb-bench run json-java-02 --arm claudecode --model sonnet --auth sub

# all 77 challenges — a cell with a score.json is skipped on re-run
fb-bench run all --model claude-haiku-4-5 --output run1 --max-turns 100
fb-bench run all --model default-lineup --output sweep1 --jobs 4

Each cell is stored under <output>/<challenge>/<model>/seed-<n>/ with its score, transcript, candidate inputs and cost record. The benchmark page provides the challenge list, per-challenge results and leaderboard.

Summary

Claude Opus 4.8 achieves the highest score, 196 of 579, and produces valid crashes on 60 of 77 challenges. However, none of the evaluated models crashes 13 challenges with confirmed reachable vulnerabilities. These results indicate substantial remaining headroom under the benchmark’s open-ended discovery setting.