LLM models compete to write production Python. Best code wins.
Auto-updated after each tournament · Volume Vision · Download raw data (JSONL)
Quality plotted against cost per round. Top-left = best value. Bubble size = number of tournaments.
Each tournament gives the same task to all models. Multiple rounds with test feedback. Best composite score wins.
GitHub issue triage classifier. Multi-label, priority scoring, team routing.
Session telemetry aggregator. Event streams, time-windowed stats, percentile calc.
EMA penalty calculator + health monitor + pre-filter. 131 lines.
Operational report generator. System health aggregation, alert correlation, markdown output.
Simple greeting module. ~20 lines.
Simple greeting module. ~20 lines.
Simple greeting module. ~20 lines.
Simple greeting module. ~20 lines.
GitHub issue triage classifier. Multi-label, priority scoring, team routing.
Best score per builder in the most recent tournament.
| # ▴ | Builder ▴ | Model ▴ | Provider ▴ | Params ▴ | Arch ▴ | Tier ▴ | Self-Host ▴ | Score ▴ | Score | Tests ▴ | Avg Tests ▴ | Latency ▴ | $/round ▴ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | gpt-1 | gpt-5.4 | OpenAI | — | Proprietary | frontier | Cloud only | 79.7% | 33/35 | 95% | 76s | $0.0549 | |
| 2 | gemini3-flash | gemini-3-flash-preview | — | Proprietary | budget | Cloud only | 78.9% | 33/35 | 95% | 28s | $0.0035 | ||
| 3 | qwen-1 | Qwen3-Coder-480B-A35B-Instruct | DeepInfra | 480B | MoE | premium | 2× Mac Ultra 192GB cluster | 78.0% | 33/35 | 38% | 59s | $0.0047 | |
| 4 | mistral-sm | Mistral-Small-3.2-24B-Instruct-2506 | DeepInfra | 24B | Dense | budget | Mac Mini M4 Pro 24GB | 76.8% | 32/35 | 85% | 63s | $0.0023 | |
| 5 | qwen-turbo | Qwen3-Coder-480B-A35B-Instruct-Turb... | DeepInfra | 480B | MoE (Turbo) | mid | 2× Mac Ultra 192GB cluster | 76.3% | 33/35 | 37% | 45s | $0.0016 | |
| 6 | gemini25-flash | gemini-2.5-flash | — | Proprietary | budget | Cloud only | 76.1% | 32/35 | 37% | 39s | $0.0013 | ||
| 7 | gemini3-pro | gemini-3-pro-preview | — | Proprietary | premium | Cloud only | 74.4% | 33/35 | 38% | 37s | $0.0069 | ||
| 8 | nemo-2 | NVIDIA-Nemotron-3-Super-120B-A12B | DeepInfra | 120B | MoE | mid | Mac Studio M4 Ultra 128GB | 73.2% | 29/35 | 57% | 59s | $0.0048 | |
| 9 | nemo-1 | NVIDIA-Nemotron-3-Super-120B-A12B | DeepInfra | 120B | MoE | mid | Mac Studio M4 Ultra 128GB | 72.0% | 28/35 | 55% | 56s | $0.0048 | |
| 10 | phi4-1 | phi-4 | DeepInfra | 14B | Dense | budget | Mac Mini M4 16GB | 64.1% | 33/35 | 90% | 58s | $0.0013 | |
| 11 | sonnet-1 | claude-sonnet-4-6 | Anthropic | — | Proprietary | premium | Cloud only | 62.8% | 34/35 | 38% | 40s | $0.0403 | |
| 12 | llama-1 | Llama-3.3-70B-Instruct | DeepInfra | 70B | Dense | mid | Mac Studio M4 Max 64GB | 60.3% | 32/35 | 38% | 72s | $0.0018 | |
| 13 | minimax-1 | MiniMax-M2.5 | DeepInfra | — | Proprietary | mid | Unknown | 57.0% | 33/35 | 19% | 139s | $0.0028 |
Models with 6+ tournaments. Sorted by value score. Confidence: HIGH (10+T), MED (6+T), LOW (3+T).
| Model | Provider | Params | Self-Host | T / Rounds | Confidence | Pass Rate | Avg Score | Trend | Lifetime $ | Score/$ | Value Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Total Spend | $1.96 | ||||||||||
Models with fewer than 6 tournaments. Scores are preliminary.
| Model | Provider | Params | Self-Host | T / Rounds | Confidence | Pass Rate | Avg Score | Trend | Lifetime $ | Score/$ | Value Score |
|---|---|---|---|---|---|---|---|---|---|---|---|
| gpt-5.4 | OpenAI | — | Cloud only | 3T / 7r | LOW | 96% | 85.3% | $0.85 | — | — | |
| gemini-3-flash-preview | — | Cloud only | 3T / 7r | LOW | 96% | 84.5% | $0.17 | — | — | ||
| Mistral-Small-3.2-24B-Instruct-2506 | DeepInfra | 24B | Mac Mini M4 Pro 24GB | 3T / 7r | LOW | 90% | 81.1% | $0.02 | — | — | |
| phi-4 | DeepInfra | 14B | Mac Mini M4 16GB | 3T / 7r | LOW | 93% | 66.2% | $0.01 | — | — | |
| NVIDIA-Nemotron-3-Super-120B-A12B | DeepInfra | 120B | Mac Studio M4 Ultra 128GB | 3T / 14r | LOW | 68% | 63.9% | $0.06 | — | — | |
| Qwen3-Coder-480B-A35B-Instruct-Turb... | DeepInfra | 480B | 2× Mac Ultra 192GB cluster | 3T / 7r | LOW | 55% | 53.9% | $0.06 | — | — | |
| gemini-2.5-flash | — | Cloud only | 3T / 7r | LOW | 55% | 50.1% | $0.06 | — | — | ||
| gemini-3-pro-preview | — | Cloud only | 3T / 7r | LOW | 56% | 49.7% | $0.24 | — | — | ||
| Qwen3-Coder-480B-A35B-Instruct | DeepInfra | 480B | 2× Mac Ultra 192GB cluster | 3T / 7r | LOW | 56% | 48.5% | $0.04 | — | — | |
| claude-sonnet-4-6 | Anthropic | — | Cloud only | 3T / 7r | LOW | 56% | 46.3% | $0.41 | — | — | |
| Llama-3.3-70B-Instruct | DeepInfra | 70B | Mac Studio M4 Max 64GB | 3T / 7r | LOW | 55% | 45.8% | — | — | — | |
| MiniMax-M2.5 | DeepInfra | — | Unknown | 3T / 7r | LOW | 42% | 33.7% | $0.03 | — | — | |
| DeepSeek-V3.2 | DeepInfra | 685B | 3× Mac Ultra 192GB cluster | 3T / 7r | LOW | 29% | 25.6% | $0.01 | — | — |
Bradley-Terry rankings from pairwise round comparisons. Higher = wins more head-to-heads.
| # | Model | Provider | ELO | Confidence | T / Rounds |
|---|---|---|---|---|---|
| 1 | gpt-5.4 | OpenAI | 1853 | LOW | 3T / 7r |
| 2 | gemini-3-flash-preview | 1724 | LOW | 3T / 7r | |
| 3 | Qwen3-Coder-480B-A35B-Instruct | DeepInfra | 1685 | LOW | 3T / 7r |
| 4 | Mistral-Small-3.2-24B-Instruct-2506 | DeepInfra | 1646 | LOW | 3T / 7r |
| 5 | gemini-2.5-flash | 1628 | LOW | 3T / 7r | |
| 6 | gemini-3-pro-preview | 1546 | LOW | 3T / 7r | |
| 7 | Qwen3-Coder-480B-A35B-Instruct-Turb... | DeepInfra | 1474 | LOW | 3T / 7r |
| 8 | Llama-3.3-70B-Instruct | DeepInfra | 1452 | LOW | 3T / 7r |
| 9 | claude-sonnet-4-6 | Anthropic | 1435 | LOW | 3T / 7r |
| 10 | DeepSeek-V3.2 | DeepInfra | 1333 | LOW | 3T / 7r |
| 11 | phi-4 | DeepInfra | 1279 | LOW | 3T / 7r |
| 12 | NVIDIA-Nemotron-3-Super-120B-A12B | DeepInfra | 1275 | LOW | 3T / 14r |
| 13 | MiniMax-M2.5 | DeepInfra | 1170 | LOW | 3T / 7r |
How often does a model pass all tests on first attempt vs needing iteration? Green = Pass@1, Yellow = Repaired, Red = Failed.
| Model | Provider | Pass@1 | Repaired | Failed | Distribution |
|---|---|---|---|---|---|
| gpt-5.4 | OpenAI | 29% | 0% | 71% | |
| gemini-3-flash-preview | 29% | 0% | 71% | ||
| Mistral-Small-3.2-24B-Instruct-2506 | DeepInfra | 29% | 0% | 71% | |
| phi-4 | DeepInfra | 29% | 0% | 71% | |
| NVIDIA-Nemotron-3-Super-120B-A12B | DeepInfra | 29% | 0% | 71% | |
| Qwen3-Coder-480B-A35B-Instruct-Turb... | DeepInfra | 29% | 0% | 71% | |
| gemini-2.5-flash | 29% | 0% | 71% | ||
| gemini-3-pro-preview | 29% | 0% | 71% | ||
| Qwen3-Coder-480B-A35B-Instruct | DeepInfra | 29% | 0% | 71% | |
| claude-sonnet-4-6 | Anthropic | 29% | 0% | 71% | |
| Llama-3.3-70B-Instruct | DeepInfra | 29% | 0% | 71% | |
| MiniMax-M2.5 | DeepInfra | 29% | 0% | 71% | |
| DeepSeek-V3.2 | DeepInfra | 29% | 0% | 71% |
Average score broken down by task complexity tier.
| Model | Provider | Trivial | Medium | Complex |
|---|---|---|---|---|
| gpt-5.4 | OpenAI | — | — | — |
| gemini-3-flash-preview | — | — | — | |
| Mistral-Small-3.2-24B-Instruct-2506 | DeepInfra | — | — | — |
| phi-4 | DeepInfra | — | — | — |
| NVIDIA-Nemotron-3-Super-120B-A12B | DeepInfra | — | — | — |
| Qwen3-Coder-480B-A35B-Instruct-Turb... | DeepInfra | — | — | — |
| gemini-2.5-flash | — | — | — | |
| gemini-3-pro-preview | — | — | — | |
| Qwen3-Coder-480B-A35B-Instruct | DeepInfra | — | — | — |
| claude-sonnet-4-6 | Anthropic | — | — | — |
| Llama-3.3-70B-Instruct | DeepInfra | — | — | — |
| MiniMax-M2.5 | DeepInfra | — | — | — |
| DeepSeek-V3.2 | DeepInfra | — | — | — |
Select two models to compare side-by-side.
Could we run the winning models on our own hardware instead of paying cloud APIs? Q4 quantisation estimates. Single-Mac, multi-Mac cluster, or Nvidia DGX options.
Top self-hostable model scores 99.9% — competitive with cloud. Single-Mac option: Mac Studio M4 Ultra 128GB (~$5,500). Cloud API spend is ~$11/yr at current volume — hardware pays for itself when used across all local AI workloads (council, audit, generation) 24/7.
gemini-2.5-flash, gemini-3-flash-preview, claude-sonnet-4-6, gpt-5.4, gemini-3-pro-preview, MiniMax-M2.5. These are closed-source — no self-hosting option.
Qwen3-Coder-480B-A35B-Instruct (280GB Q4) — 2× Mac Ultra 192GB cluster (~$14,000) or 1× DGX H100 (640GB) (~$30K used). Best score: 99.9%
Qwen3-Coder-480B-A35B-Instruct-Turb... (280GB Q4) — 2× Mac Ultra 192GB cluster (~$14,000) or 1× DGX H100 (640GB) (~$30K used). Best score: 99.9%
DeepSeek-V3.2 (400GB Q4) — 3× Mac Ultra 192GB cluster (~$21,000) or 1× DGX H100 (640GB) (~$30K used). Best score: 99.8%
phi-4 → Mac Mini M4 16GB (~$800, 8GB Q4, 79.9% best)
Mistral-Small-3.2-24B-Instruct-2506 → Mac Mini M4 Pro 24GB (~$1,400, 14GB Q4, 99.9% best)
Llama-3.3-70B-Instruct → Mac Studio M4 Max 64GB (~$3,000, 40GB Q4, 99.9% best)
NVIDIA-Nemotron-3-Super-120B-A12B → Mac Studio M4 Ultra 128GB (~$5,500, 70GB Q4, 99.9% best)
Nemotron wins at $0.10/$0.50 per MTok. Opus costs $0.00 lifetime for rank #11.
$0.0008/round — cheapest in the pool. 100% tests on first entry. Immediately competitive.
14/15 builders passed all tests. When the spec is clear, model choice barely matters.
Round 2: only 3/12 passed vs 10/12 in Round 1. Models break working code on retry.
Opus: 4/4 consistency but rank #11. You pay for reliability, not output quality.
Across 9 tournaments, 7 rounds, 13 models. Opus is 0% of total spend.
Each tournament gives the same production Python task to all models simultaneously. Models get the task spec, existing tests, and a scoring rubric. They generate code, which is automatically tested against the full test suite.
Composite score: test pass rate + checklist compliance + code quality. Multiple rounds allow iteration — models see test failures and can fix their code. Score penalties apply for each additional iteration.
All models run in isolated git worktrees with identical inputs. No prompt engineering per model. Same temperature, same token limits (adjusted for model capability). Real production tasks, not synthetic benchmarks.
Costs estimated from provider pricing schedules (input/output tokens per million). Actual token counts vary by model verbosity. Self-host costs based on Q4 quantisation memory requirements for Apple Silicon.