Model selection
Head-to-head comparisons across providers and sizes on your workload, weighing quality, latency and cost together.
Practice 03 — Model testing & evals
Benchmarks don't run your business. Your tasks do.
We build evaluation harnesses on your own data and success criteria, so choosing a model, changing a prompt or upgrading a version is a measured decision — not a vibe check.
deepswe · 113 tasksoct 2026
| model | harness | pass@1 | $/task |
|---|---|---|---|
| GPT-5.6 Luna | Copilot | 74.1% | $0.37 |
| Gemini 3.8 Flash | Copilot | 74.1% | $2.79 |
| Claude Opus 5 | Copilot | 71.2% | $5.66 |
| GPT-6 Astra | Copilot | 71.0% | $8.43 |
| Claude Opus 5.5 | Copilot | 70.8% | $11.98 |
| Claude Opus 5 | Claude Code | 64.4% | $13.49 |
| Auto | Copilot | 37.0% | $1.62 |
What we test
Golden datasets
Curated, versioned test sets labelled with your domain experts — the ground truth every change is measured against.
Graders you can trust
Code checks where possible, model-graded rubrics where needed — each calibrated against human judgment.
Agent & tool-use evals
Multi-step trajectory testing: did the agent call the right tools, recover from errors and stop when it should?
Safety & red-teaming
Adversarial suites for prompt injection, data leakage and policy violations before your users find them.
Production monitoring
Online evals and sampling that catch drift when inputs, models or upstream data change underneath you.
From our own bench
16 agent configurations. 7,500+ attempts at real engineering tasks.
We ran today's frontier models through DeepSWE — 113 real-world software tasks, each attempted four or five times — and graded every attempt against its tests. The leaderboard wasn't the interesting part.
Same model, different harness, different result.
Claude Opus 5 passed 71.2% of attempts inside GitHub Copilot and 64.4% inside Claude Code — and cost 58% less per task ($5.66 vs $13.49).
The cheapest model tied for first.
GPT-5.6 Luna passed 74.1% at $0.37 per task — about 1/32 the cost of Claude Opus 5.5, which scored 70.8%.
Version bumps move the numbers.
Luna fell from 74.1% to 69.9% between two Copilot CLI releases with no change to the model. That is the regression an eval gate catches before rollout.
Results that hold up.
Where our configurations overlap Meta's published DeepSWE evaluation, our pass counts land within about three passes in 565 attempts.
When teams call us
You can't answer “is it better?” with confidence.
- A new model launched and you don't know if switching is safe.
- Prompt tweaks fix one case and quietly break three others.
- Leadership wants proof before a wider rollout.
- Your AI spend is climbing and nobody knows what a cheaper model would cost you in quality.
- A regulator, auditor or customer is asking how you test.
Typical engagement
We build the harness alongside your team and wire it into CI, so evals keep running long after we've left.
- DURATION
- [WEEKS]
- SCOPE
- [USE CASES]
- YOU GET
- Eval harness, golden sets, CI gates, model recommendation
- INVESTMENT
- [YOUR PRICE]