Why we built it

Language models are claimed to find software vulnerabilities, to read a codebase and surface its bugs. Vendors demo it, papers report it, and security teams are starting to buy it. There is a problem underneath all of it. We cannot verify the claim, because no public benchmark measures it.

Look at what the existing benchmarks actually ask a model to do. Some ask it to label a function it never runs. Some ask it to write new code and then scan the output. Some ask it to patch a bug that is handed over, or to re-trigger a bug it was already described. Not one of them asks the model to find a bug it was not told about and prove that the bug actually fires. That is the capability the industry is selling, and it is exactly the one that goes unmeasured.

What makes it a bug-finding benchmark

A benchmark measures bug-finding, not labeling, reproduction, or repair, only if three properties hold at the same time. FuzzingBrain-Bench is designed to hold all three.

🧬 Inside the corpus

The 68 bugs are not toy cases. Every one is a real, already-disclosed fault in production open-source software, validated by a golden PoC, and reproduced through an audited public-API entry point. Here is the whole corpus, by the numbers.

🌐 Three languages, forty projects

Heterogeneity is deliberate. A single-language set lets a model overfit one parser family, so the corpus spans C, C++, and the JVM, from image codecs and font shapers to protocol stacks, compilers, and document parsers.

💥 Seventeen fault classes

The bench is not a wall of the same overflow. It covers memory corruption, memory disclosure, undefined behavior, resource exhaustion, and JVM exceptions, each built with the sanitizer it was originally found with (ASan, UBSan, LSan, libFuzzer OOM and timeout, Jazzer).

⚠️ Severity, grouped by impact

Reading the same 68 bugs by what an attacker gets, the corpus leans toward the classes that matter most. Memory corruption, the band that yields write primitives, is the single largest group.

🎚️ A calibrated difficulty spectrum

A good benchmark is neither saturated nor impossible. We graded every bug in the hardest blind mode, full-scan, with two frontier models, and the corpus spreads cleanly across the difficulty range. Nothing is free, and nothing is hopeless.

🪜 Every attempt is graded on a five-rung ladder

A single crash-or-not bit throws away most of the signal. Instead each candidate input climbs a monotone ladder, and each rung is strictly stronger than the last. A crash of the wrong class or the wrong location is a different bug, and it fails.

⚡ How to use it

The whole benchmark is a public, answer-free Docker image per bug plus a remote grading oracle that holds the answer key. You need only Docker and a model API key. Nothing you clone contains the PoC, the expected fault, or the fixed build. Those live only behind the oracle, which returns a verdict and nothing else.

Setup

git clone https://github.com/fuzzingbrain/FuzzingBrain-Bench
cd FuzzingBrain-Bench

python3 -m venv .venv && source .venv/bin/activate
pip install -e .                      # Python 3.10+ and Docker

# drop your key(s) into ./.env, auto-loaded on every run
cat > .env <<'EOF'
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
EOF

fb-bench list                         # the 68 challenges, by neutral alias
fb-bench models                       # supported models and which keys are loaded

Run one challenge with a model

The runner pulls the public challenge image, drives the agent loop over a six-tool MCP interface (setup, read_file, list_directory, write_file, exec, grade), and grades candidates against the remote oracle.

# normal mode, harness plus source plus a neutral description
fb-bench run avro-03 --model claude-opus-4-8

# full-scan, the hardest public mode, no description, find the crash cold
fb-bench run avro-03 --model gpt-5.5 --full-scan --max-turns 100

Run the whole corpus

# one model over all 68 challenges, resumable, rerun to skip done cells
python -m fbbench.sweep.orchestrator --models claude-opus-4-8 --bugs all --exp run1

# default multi-model lineup, all challenges
python -m fbbench.sweep.orchestrator --models sweep --bugs all --exp sweep1

Each cell lands in runs/<exp>/<bug>/<model>/run-N/ with the full ladder verdict (score.json), the per-turn transcript, a distilled trajectory, and cost. A leaderboard prints at the end and is re-aggregable with --report-only.

Grade any external PoC, no LLM required

The oracle is vendor-neutral. Hand it a blob from AFL++, libFuzzer, honggfuzz, or your own hands, and it returns the capability ladder.

fb-bench grade avro-03 my-input.bin        # -v for the per-rung evidence

📊 What frontier models actually score

We ran two frontier models, Claude Haiku 4.5 and GPT-5.5, across all six modes on 68 bugs, for 816 episodes. The per-mode breakdown along the ladder is below. Three findings stand out.

Finding a bug is harder than reproducing one

With the full description (normal), GPT-5.5 solves 58/68, which is 85 percent. Remove the description and the best blind full-scan rate drops to 71 percent, and for Haiku it falls much harder, from 57 percent to somewhere between 25 and 34 percent. The description is worth about 14 points to GPT-5.5 and about 23 to Haiku, a direct measurement of how much of the apparent skill was reading comprehension.

Did it crash over-counts solves by about a quarter

Across all modes, 525 cells produced a crash, but only 75 percent were the preset vulnerability. 12 percent were off-target, a different fault class, and 14 percent hit the right class at the wrong location. A benchmark that reports only whether the binary crashed therefore overstates bug-finding by a quarter. The class and site rungs are exactly what separate found a bug from found the bug.

Difficulty is trigger-construction, not depth

One might expect a bug buried deep below the harness to be harder. It is not. The correlation between call-chain depth and solvability is essentially zero. What defeats the models is constructing the precise triggering input, which is independent of how deep the site sits. This is why full-scan is about equal to diff-scan even though the latter reveals the file. File-level localization was never the bottleneck.

The bottom line

FuzzingBrain-Bench measures real-world bug-finding, graded by the program itself, on a ladder, with off-target crashes thrown out, across 68 real bugs, 40 projects, 17 fault classes, and a calibrated difficulty spectrum. Its verdict on the current frontier is sober and specific. Models cannot yet reliably find real bugs in real software. Finding is harder than reproducing, did it crash over-counts by a quarter, and difficulty lives in trigger-construction, not depth.

It is public, deterministic, single-attempt, and reproducible. Clone it, point your favorite model at it, and see for yourself.