FuzzingBrain

Run it in five minutes

Only Docker and a model key are needed. No build, no answer key, nothing to download but the sealed image.

Works with

1. Install

git clone https://github.com/fuzzingbrain/FuzzingBrain-Bench
cd FuzzingBrain-Bench && pip install -e .

2. Run a bug through the model API

The direct arm. Pick your provider, the key goes in ./.env, then swap --model for any model from the same provider.

echo "ANTHROPIC_API_KEY=sk-ant-..." >> .env
fb-bench run json-java-02 --model claude-haiku-4-5
# swap: claude-sonnet-4-6, claude-opus-4-7

echo "OPENAI_API_KEY=sk-..." >> .env
fb-bench run json-java-02 --model gpt-5.4-mini
# swap: gpt-5, gpt-5.4, gpt-5.5

echo "GEMINI_API_KEY=..." >> .env
fb-bench run json-java-02 --model gemini-2.5-flash
# swap: gemini-2.5-pro, gemini-3.5-flash, gemini-3.1-pro-preview

echo "DEEPSEEK_API_KEY=sk-..." >> .env
fb-bench run json-java-02 --model deepseek-v4-flash
# swap: deepseek-v4-pro

3. Or run a real coding agent

The agent arms drive a full coding-agent CLI through the same bench MCP server, so the model plans, runs its own tools, and iterates. These are the runs behind the agent leaderboard.

# one-time: install the codex CLI and log in with an API key (not a ChatGPT login)
npm install -g @openai/codex
printenv OPENAI_API_KEY | codex login --with-api-key

# run one bug, or sweep the whole corpus (resumable)
python -m fbbench.sweep.codex one json-java-02
python -m fbbench.sweep.codex sweep --bugs all

# one-time: install the claude CLI
npm install -g @anthropic-ai/claude-code

# subscription login (default), uses your claude.ai plan
python -m fbbench.sweep.claudecode one json-java-02 --model sonnet

# or an API key from ./.env (pay as you go, no session limit)
python -m fbbench.sweep.claudecode sweep --bugs all --model sonnet --auth api

4. Sweep the whole corpus

The model-API arm across all 68 bugs. Each cell lands under runs/<bug>/<model>/. The agent arms above have their own resumable sweep.

python -m fbbench.sweep.orchestrator --models claude-haiku-4-5 --bugs all --exp run1

Some models to start with

A partial list, not everything supported. Flagship is strongest, fast is cheapest. Any other model from these four providers works too, routed automatically by its name.

Challenge list

Every challenge is a real, publicly known bug that FuzzingBrain found in an open-source library. There are 68 across 40 projects, each shipped as a sealed image under a neutral alias. Project and language are public. The fault, its crash site, and the required levels stay sealed.

Click a column to sort — by difficulty, language, project or id. The coloured Difficulty badge runs 1 easiest to 5 hardest, the dynamic ranking (D = 5 − ⌈solves/2⌉ over the eight runs) from the Difficulty section →. The 17 5 bugs are the hard tail no run has cracked yet.

Bug categories

Every verdict is decided by one of five oracles, the sanitizer or engine that judges the run. The bugs themselves span memory-safety faults and denial-of-service classes.

The oracles

By category

Difficulty

A dynamic ranking. Each bug is scored by how many frontier model runs actually solve it, then placed into one of five levels.

The N = 8 runs are the frontier models and agents on the leaderboard →, each a single full-scan pass over all 68 bugs. Because the score depends on the current lineup, difficulty is not fixed — it drops as the models improve. We tally S for every bug, apply the formula, and the counts fall out as the pyramid below: a broad middle with a hard tail of 17 bugs no run has cracked yet.

In one line: a bug's difficulty is D = 5 − ⌈S / 2⌉, where S is how many of the 8 model runs fully solved it — so D1 is a bug nearly everyone cracks and D5 is one no run has solved yet.

How far each model climbs

The same five difficulty tiers, now split by model: how many bugs each run solved at each level. Everyone clears the easy floor, nobody cracks the D5 tail, and the field fans out in the middle — that is where the score is really won.

Solve rate by vulnerability category

Pooled over the eight model runs, broken out by fault category and by language. One pattern runs through it: a bug that fires on a single malformed input is easy, while a bug that needs the model to construct a condition — free-then-use, exhaust memory, leak — is hard.

Categories shown faint have only 1–4 bugs — treat those rates as indicative, not precise. The large categories (out-of-bounds-read 22, memory-exhaustion 9, uncaught-exception 8, null-pointer-dereference 6) are the solid signal.

By language

JVM bugs are mostly uncaught exceptions a single bad field triggers; C is all low-level memory faults where the input must be built byte by byte.

Leaderboard

Ranked by difficulty-weighted score on all 68 bugs, one full-scan pass per model. A solve means the run found the documented bug on its own and cleared all five ladder rungs — reach, crash, differential, class and site — so off-target crashes do not count. Time is the median per bug; turns and grade calls are per-bug means, and cost is dollars per bug and per solved bug. Click any column to sort.

Knowing when to stop is part of the task. A real bug hunt may turn up nothing, so a model is free to give up when it finds no fault — and the Turns column shows it. The small pair is mean turns when the model solved · when it gave up. For every API model a give-up costs about twice the turns of a solve (Opus 20·54, GPT 20·33, Haiku 40·80): it keeps digging, then stops. The two agents never self-terminate, so both read 100·100. Cost, though, follows the per-turn price, not the turn count — ¢/turn spans 15× (Opus 8.5¢ vs DeepSeek Flash 0.6¢), so a cheap model that burns many turns still lands cheap.

One pass per model (seed 0). Agent rows wrap the same base model in a coding agent (Codex, Claude Code); API rows call the model directly. Grade calls are shown for API runs only — the agent harnesses do not expose reliable per-tool telemetry, so those cells read —. Gemini and further models are being added. See how each model’s score breaks down by difficulty →