CWE-bench v2 Opens 100 "Find and Fix the Vulnerability" Tasks Covering 91 Weakness Types: Among the 11 Models Collinear Tested, Gemini 4 Argon Passes 67.8% on the First Try, and 19% of Tasks Still Fail Every Time

What it tests

AI is making vulnerabilities faster to find, which turns the defender's question into whether coding agents can actually patch them in real code. CWE-bench v2, released by Collinear AI on Hugging Face on November 3, breaks that into 100 public tasks: an agent gets a checkout of a real open-source repository and a short instruction, and must find and fix a security weakness without breaking existing functionality.

A few design choices stand out:

• Coverage: the 100 tasks span 91 CWE (Common Weakness Enumeration) types. Collinear's reasoning is that a defender has to anticipate every way an attacker could get in, so recognizing familiar patterns isn't enough.
• Scoring: fully programmatic, deterministic and reproducible at no extra cost; the checks were refined with the help of LLM judge panels to be both hard and fair.
• Quality control: every task was checked on two points: that the instruction reads like a realistic request rather than an evaluation, and that the verifier tests for correct behavior rather than the structure of one particular fix. Every task was also reviewed by hand.
• Held-out set: a separate private set tracks progress on tasks no model has seen; evaluating on it requires contacting Collinear.

Results for 11 models

In Collinear's published results for 11 models, Gemini 4 Argon scored 67.8% pass@1 (first-attempt pass rate) and 81.0% pass@4 (at least one pass in four attempts), with Claude Opus 5.5 just behind at 67.3% and 78.0%. Of the 100 tasks, 95 were solved by at least one model, showing the tasks are solvable; but even the top-scoring model failed 19% of the set on every attempt.

On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the cost-versus-score Pareto frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds about ten points at $1.83, and Gemini edges slightly higher at $3.92. Open-weight results on the dataset page's leaderboard update as the community submits PRs, and the full comparison, including API models such as Gemini, Claude and GPT, lives in a separate Space.

How to use it

Each task is a self-contained Harbor task (a Docker environment, an instruction and a deterministic verifier), and the dataset ships a registry.json, so harbor run can launch the whole benchmark straight from the Hugging Face repository; -k 4 runs four attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated, so you have to accept its access conditions first. Tasks are also tagged as reinforcement learning environments, meaning the same environments and verifiers can provide rewards for training defensive agents.

These numbers apply only to these 100 public tasks and the 11 models Collinear ran; a different task set or evaluator could produce a different order. For choosing a model, the more useful signals are that the leading models still fail roughly a fifth of the tasks every time, and that DeepSeek V4.1 Flash and Opus 5.5 are about ten points apart at roughly a 45-fold difference in cost per rollout.