AI Agent Evaluations
Evaluations of AI coding agents challenges for Nitro
Last updated September 4, 2026.
| Model | Agent | Avg Duration | Avg List Cost | Success Rate | Success Rate with AGENTS.md | First-Try Rate |
|---|---|---|---|---|---|---|
| GPT 5.6 Sol (xhigh) | Codex | 1m 31s | $0.126 | 100.0% (41/41) | 100.0% (±0.0) | 100.0% |
| Claude Opus 5 | Claude Code | 2m 20s | $0.682 | 100.0% (41/41) | 100.0% (±0.0) | 100.0% |
| Claude Fable 5.1 | Claude Code | 2m 14s | $0.578 | 100.0% (41/41) | 100.0% (±0.0) | 100.0% |
| Gemini 3.7 Flash | OpenCode | 5m 35s | $0.284 | 97.6% (40/41) | 100.0% (+2.4) | 97.6% |
| Claude Sonnet 5 | Claude Code | 1m 49s | $0.300 | 97.6% (40/41) | 100.0% (+2.4) | 97.6% |
| Kimi K3 | OpenCode | 3m 29s | $0.272 | 97.6% (40/41) | 100.0% (+2.4) | 90.2% |
| GLM 5.3 | OpenCode | 4m 17s | $0.127 | 97.6% (40/41) | 100.0% (+2.4) | 90.2% |
| Cursor Composer 2.5 | Cursor | 1m 18s | $0.107 | 97.4% (38/39) | 100.0% (+2.6) | 97.4% |
| Kimi K2.7 Code | OpenCode | 2m 22s | $0.132 | 95.1% (39/41) | 97.6% (+2.4) | 87.8% |
| Gemini 3.8 Flash | OpenCode | 4m 54s | $0.399 | 92.7% (38/41) | 97.6% (+4.9) | 70.7% |
| Gemini 3.1 Pro Preview | OpenCode | 4m 05s | $0.377 | 90.2% (37/41) | 100.0% (+9.8) | 85.4% |
| MiniMax M3 | OpenCode | 2m 05s | $0.049 | 90.2% (37/41) | 95.1% (+4.9) | 73.2% |
| Claude Haiku 4.5 | Claude Code | 1m 09s | $0.192 | 43.9% (18/41) | 78.0% (+34.1) | 29.3% |