Paweł Huryn has published the website and data for his new Bug Hunt Bench, Paweł's benchmark that tests whether coding models can locate and repair bugs in real repositories. Its leaderboard was updated on September 4, 2026, and the accompanying GitHub repository is open for stars.
The benchmark uses two repositories containing 105 planted bugs. Models must both identify a defect and submit a fix. Another model judges the proposed fixes, which are then matched against a withheld answer key.
A suspected bug earns no points unless the model fixes it. Huryn identifies “suspected, not confirmed” as the most common failure case. The score counts only planted bugs that were fixed and verified; real defects that were not planted are tracked separately and excluded from the benchmark score.
GPT-6 Astra leads the initial leaderboard
The leaderboard includes 21 of 41 runs and is ranked by the number of planted bugs fixed. GPT-6 Astra Max currently leads with 48 of 105 fixes.
| Model configuration | Planted bugs fixed | Unplanted defects | Runtime | Estimated cost |
|---|---|---|---|---|
| GPT-6 Astra Max | 48 | 45 | 78.8 minutes | $31.21 |
| Fable 5.1 Max | 43 | 11 | 73.1 minutes | $77.55 |
| GPT-5.6 Sol Max | 42 | 40 | 163.8 minutes | $69.61 |
| GPT-5.6 Sol High | 34 | 28 | 66.7 minutes | $33.92 |
| GPT-5.6 Luna Max | 33 | 31 | 85.7 minutes | $1.80 |
| Fable 5.1 High | 33 | 7 | 36.3 minutes | $41.52 |
| Fable 5 Max | 29 | 5 | 57.3 minutes | $104.49 |
| Grok 4.6 XHigh | 27 | 16 | 41.1 minutes | $16.96 |
| Opus 5 Max | 27 | 2 | 60 minutes | $51.33 |
| Fable 5 High | 24 | 3 | 31.4 minutes | $68.07 |
The initial comparison places GPT-6 Astra Max ahead of Fable 5.1 Max and GPT-5.6 Sol Max on fixes, although it also recorded more unplanted defects than either of those models. Fable 5.1 High completed its run faster than the leading configurations, while GPT-5.6 Luna Max posted a much lower estimated cost but fixed 33 planted bugs.
Lower-ranked runs include Kimi K3 Default with 21 fixes, Gemini 3.8 Flash High with 20, GLM-5.3 Default and Qwen3.8-Max XHigh with 19 each, and DeepSeek V4-Flash Default with 14. Sonnet 5 High and Opus 4.8 High each fixed nine planted bugs.
Results depend on more than bug counts
The benchmark records runtime, estimated cost and unplanted defects alongside the main score. GPT-5.6 Sol Max, for example, fixed 42 planted bugs but also produced 40 fixes for defects that were not part of the planted set. Opus 5 Max fixed 27 planted bugs and recorded two unplanted defects, while DeepSeek V4-Flash Default recorded no unplanted defects but fixed 14 planted bugs.
The listed configurations use tools including OpenAI Codex CLI, Anthropic Claude Code, Grok Build CLI, Antigravity CLI, OpenRouter and Alibaba API, either individually or in combination. Cost figures are generally list-rate estimates. The benchmark notes that GLM-5.3, Hy4 Preview, DeepSeek V4-Flash and GLM-5.3 Flash use real bills, while Grok 4.6’s figure is a reconstructed lower bound because its CLI reports context occupancy rather than billing.
The benchmark is presented as an early dataset rather than a final ranking. Its results measure successful identification and repair of planted defects, while the separate unplanted-defect count provides additional information about what each run changed beyond the target set.
Source: Paweł Huryn on X
