The harness is part of the product.
Claude Opus 5 passed 71.2% of attempts inside GitHub Copilot and 64.4% inside Claude Code, at $5.66 versus $13.49 per attempt. Same model; different tools, context handling and prompts around it.
Research — Wharf / DeepSWE
What 7,571 coding-agent attempts reveal.
Wharf is our benchmark platform for AI coding agents: real software tasks, repeated attempts, an independent verifier, and every result retained. We pointed it at today's frontier models and harnesses.
- TASKS
- 113
- CONFIGURATIONS
- 16
- ATTEMPTS
- 7,571
- ATTEMPTS PER TASK
- 4–5
Key findings
Three things a leaderboard doesn't show.
More expensive isn't more effective.
GPT-5.6 Luna passed 74.1% at $0.37 per attempt. Claude Opus 5.5 passed 70.8% at $11.98. Routing is no shortcut either: Copilot's Auto router passed 37.0%, its HydraFusion router 70.4%.
Passing the task isn't enough.
GPT-6 Astra and Claude Opus 5 (Copilot) passed almost the same number of attempts — 321 and 322. But 43 of Astra's attempts broke existing tests, against 20 for Opus 5.
Results
Quality against cost, every configuration.
Each dot is a complete configuration — model, harness release and reasoning effort. Up and to the left is better. The copper line joins Claude Opus 5 under two harnesses.
| model | harness | effort | passed | pass@1 | broke tests | $/attempt | costed |
|---|---|---|---|---|---|---|---|
| GPT-5.6 Luna | Copilot 1.0.77 | max | 335/452 | 74.1% | 17 | $0.37 | 97% |
| Gemini 3.8 Flash | Copilot 1.0.91 | high | 335/452 | 74.1% | 20 | $2.79 | 100% |
| Claude Opus 5 | Copilot 1.0.77 | max | 322/452 | 71.2% | 20 | $5.66 | 99% |
| GPT-6 Astra | Copilot 1.0.91 | max | 321/452 | 71.0% | 43 | $8.43 | 99% |
| Claude Opus 5.5 | Copilot 1.0.95 | max | 320/452 | 70.8% | 22 | $11.98 | 82% |
| HydraFusion (all-1) | Copilot 1.0.98 | default | 318/452 | 70.4% | 28 | $1.21 | 100% |
| HydraFusion (all-2) | Copilot 1.0.95 | max | 317/452 | 70.1% | 32 | $1.17 | 99% |
| GPT-5.6 Luna | Copilot 1.0.91 | max | 316/452 | 69.9% | 21 | $0.37 | 99% |
| GPT-5.6 Sol | Copilot 1.0.77 | xhigh | 315/452 | 69.7% | 23 | $8.44 | 99% |
| HydraFusion (all-1) | Copilot 1.0.95 | max | 311/452 | 68.8% | 36 | $1.21 | 99% |
| Claude Opus 5.5 | Copilot 1.0.91 | max | 309/452 | 68.4% | 27 | $11.41 | 87% |
| GPT-5.6 Terra | Copilot 1.0.77 | max | 376/565 | 66.5% | 37 | $4.26 | 98% |
| Claude Opus 5 | Claude Code 2.1.220 | max | 291/452 | 64.4% | 21 | $13.49 | 99% |
| GPT-6 Luna | Copilot 1.0.91 | max | 276/452 | 61.1% | 36 | $0.18 | 100% |
| Muse Spark 1.2 | Muse Code 0.1.0 | xhigh | 338/565 | 59.8% | 44 | — | 0% |
| Auto | Copilot 1.0.77 | max | 209/565 | 37.0% | 48 | $1.62 | 95% |
pass@1 = passing attempts ÷ planned attempts. Broke tests = attempts where at least one pre-existing test failed. $/attempt = mean over attempts with cost data. Source: Wharf @ 661b6e3.
Reading the numbers
A pass rate is only the beginning.
Take five tasks, two attempts each, $4 of compute in total. The same results give four different answers depending on the question you ask.
Aren't repeated attempts just retries?
No. A production retry usually stops at the first success; a benchmark runs every planned attempt to measure consistency. And an infrastructure retry — re-running because a container died — is not a new model attempt at all. We count the three separately.
| task | attempt 1 | attempt 2 |
|---|---|---|
| Login | ✓ pass | ✓ pass |
| Tax calculation | ✗ fail | ✓ pass |
| Database | ✗ fail | ✗ fail |
| File upload | ✓ pass | ✗ fail |
| Search | ✗ fail | ✓ pass |
- ATTEMPT PASS RATE
- 50% — 5 of 10 attempts
- TASKS SOLVED
- 80% — 4 of 5 tasks, at least once
- COST PER PASSING ATTEMPT
- $0.80 — $4 ÷ 5 passes
- COST PER TASK SOLVED
- $1.00 — $4 ÷ 4 tasks
Methodology
The agent doesn't grade its own homework.
Implementation and verification are separate, so a claimed fix and a verified fix can never be confused.
- 01
Fixed task set
113 DeepSWE tasks, 4–5 planned attempts each. Every planned attempt stays in the denominator — a crash or a missing attempt counts as a failure.
- 02
Isolated agent run
Each attempt runs in its own clean environment. The agent's only output is a patch.
- 03
Independent verifier
A separate environment applies the patch and runs the task's tests. An attempt passes only on a positive verifier result; a timeout scores zero.
- 04
Regression check
Pre-existing tests run too. Breaking them is recorded separately, not hidden inside the pass rate.
- 05
Evidence retained
Patches, logs, agent trajectories, verifier output, timing, token usage and cost — for every attempt.
- These results describe 113 open-source coding tasks. Your workload may rank the same configurations differently — that's the point of testing on it.
- Harness comparisons change more than one thing at once: the harness release, its prompts and its tools all differ.
- Attempts on the same task are correlated. Gaps of a few points are within run-to-run noise: the same Luna configuration scored 74.1% and 69.9% on consecutive Copilot releases.
- Cost is missing for some attempts — 18% of the Opus 5.5 run and all of the Muse Spark run. Missing costs are excluded from averages, never counted as zero.
- Regression counts cover attempts where existing tests could be measured. Unmeasured is unknown, not clean.
- Models and harnesses ship monthly. This is a Oct 2026 snapshot.