AI Agent Evaluations
Evaluations of AI coding agents challenges for Nitro
Last updated September 4, 2026.
| Model | Agent | Avg Duration | Avg List Cost | Success Rate | Success Rate with AGENTS.md | First-Try Rate |
|---|---|---|---|---|---|---|
| GPT 5.6 Sol (xhigh) | Codex | 1m 41s | $0.140 | 100.0% (47/47) | 100.0% (±0.0) | 100.0% |
| Claude Opus 5 | Claude Code | 2m 28s | $0.746 | 100.0% (47/47) | 100.0% (±0.0) | 100.0% |
| Claude Fable 5.1 | Claude Code | 2m 19s | $0.606 | 100.0% (47/47) | 100.0% (±0.0) | 100.0% |
| Gemini 3.7 Flash | OpenCode | 5m 28s | $0.286 | 97.9% (46/47) | 100.0% (+2.1) | 97.9% |
| Kimi K3 | OpenCode | 3m 39s | $0.282 | 97.9% (46/47) | 100.0% (+2.1) | 91.5% |
| GLM 5.3 | OpenCode | 4m 48s | $0.148 | 97.9% (46/47) | 100.0% (+2.1) | 91.5% |
| Claude Sonnet 5 | Claude Code | 1m 52s | $0.313 | 95.7% (45/47) | 100.0% (+4.3) | 95.7% |
| Kimi K2.7 Code | OpenCode | 2m 28s | $0.137 | 95.7% (45/47) | 97.9% (+2.1) | 87.2% |
| Cursor Composer 2.5 | Cursor | 1m 22s | $0.112 | 95.6% (43/45) | 100.0% (+4.4) | 95.6% |
| Gemini 3.8 Flash | OpenCode | 5m 12s | $0.417 | 93.6% (44/47) | 97.9% (+4.3) | 70.2% |
| Gemini 3.1 Pro Preview | OpenCode | 4m 20s | $0.392 | 89.4% (42/47) | 100.0% (+10.6) | 85.1% |
| MiniMax M3 | OpenCode | 2m 08s | $0.054 | 89.4% (42/47) | 93.6% (+4.3) | 74.5% |
| Claude Haiku 4.5 | Claude Code | 1m 10s | $0.193 | 40.4% (19/47) | 76.6% (+36.2) | 25.5% |