AI Model Comparison (Sep 2026) β Price, Reasoning, Coding, Speed
ai/ai-model-comparison-2026 Β· Updated Β· Tags: ai, llm, benchmarks, pricing, claude, gpt, grok, glm, deepseek, cursor
AI Model Comparison (Sep 2026) β Price, Reasoning, Coding, Speed
Colour key: πͺ Claude π© OpenAI π¦ Grok π¨ GLM π§ DeepSeek β¬ Composer
TL;DR
- Default pick: Claude Opus 5 β 7.6/10 capability at $5/$25, within 0.9 points of the best at half the price.
- Capability ceiling: Claude Fable 5.1 (8.5/10) β $7.17/MTok, CP value 1.2, and slowest to first token.
- Value pick: DeepSeek V4.1-Flash β CP value 34.6, 29Γ Fable's β $0.18/MTok, cache hits at $0.006, 194 tok/s.
- Astra and Fable 5.1 tie on coding β 58.2% vs 57.9% on the official Terminal-Bench 4.0 board, inside each other's error bars.
- Give your agent docs before buying a better model β an
AGENTS.mdlifts Sonnet 5 from 81% to 97% on Next.js Evals, matching Fable 5.1 at 1/5 the cost.
Status right now β 11 Sep 2026
| Model | Vendor | Released | AA Intelligence rank | Still current? |
|---|---|---|---|---|
| Claude Fable 5.1 | Anthropic | 1 Sep 2026 | #1 / 200 (53) | β 10 days old β newest frontier model here |
| GPT-6 Astra | OpenAI | 3 Sep 2026 | #1= (53) | β 8 days old |
| Claude Opus 5 | Anthropic | 24 Jul 2026 | #7 / 200 (51) | β 7 weeks; retirement no sooner than 24 Jul 2027 |
| GLM-5.3 | Z.ai | 14 Aug 2026 | #1 open-weights (45) | β 4 weeks |
| Grok 4.6 | xAI | 12 Aug 2026 | (44) | β 4 weeks |
| DeepSeek V4.1-Flash | DeepSeek | 10 Sep 2026 | (40) | β 1 day old |
| Claude Sonnet 5 | Anthropic | 30 Jun 2026 | #42 / 200 (38) | β 10 weeks |
| Cursor Composer 2.5 | Cursor | 18 May 2026 | not listed | β οΈ 16 weeks, no successor |
- DeepSeek V4-Pro is being retired in 3 days. From 12:00 Beijing, 14 Sep 2026, every
deepseek-v4-prorequest routes to V4.1-Flash and bills at Flash prices, "until V4.1 Pro is released". If you havedeepseek-v4-propinned in code, your model changes under you this week. V4.1-Flash scores higher anyway (40 vs 36) at ~1/4 the price. - GLM-5.3 is open-weights but not MIT. Its bespoke
glm-5.3licence requires MaaS operators with >$10B revenue over any 12 consecutive months to pass a Z.ai security review. DeepSeek is plain MIT β no such clause. - Cursor Composer 2.5 has gone 16 weeks without a refresh while five competitors shipped new flagships. Cursor 3.0 (Apr 2026) and the Sep 2026 self-hosted/builds features show the product is very much alive β the model just isn't moving.
Every model here is actively served and none is an abandonment risk. Anthropic and OpenAI are trading the frontier on a ~2-week cycle; DeepSeek and Z.ai are within 8β13 index points at a fraction of the price and closing; Composer 2.5 is the only one drifting.
Summary table β the factors that decide it
Four capability axes scored 0β10, averaged into a single 0β10 capability score, against real USD prices. CP value = capability Γ· $/MTok β capability points per dollar. Sorted by CP value.
| Model | π² Input | π² Output | π² Cache hit | π² $/MTok | π§ Reason | π» Coding | π General | β‘ Speed | Capability /10 | π° CP value |
|---|---|---|---|---|---|---|---|---|---|---|
| π§ DeepSeek V4.1-Flash | $0.30 | $1.20 | $0.006 | $0.18 | 4.5 | 5 (est) | 6 | 10 | 6.4 | π 34.6 |
| π¨ GLM-5.3 | $1.40 | $4.40 | $0.26 | $0.90 | 6.5 | 7 | 7 | 4.5 | 6.2 | 6.9 |
| β¬ Composer 2.5 | $0.50 | $2.50 | $0.20 | $0.49 | 1 (est) | 2.5 (est) | 1 | 9 (est) | 3.4 | 6.9 |
| π¦ Grok 4.6 | $2.00 | $6.00 | $0.50 | $1.35 | 6 | 3.5 | 6.5 | 4 | 5.0 | 3.7 |
| πͺ Claude Sonnet 5 | $2.00 | $10.00 | $0.20 | $1.54 | 3.5 | 2 | 6.5 | 5 | 4.2 | 2.8 |
| π© GPT-5.6 Sol | $4.00 | $20.00 | $0.40 | $3.08 | 7.5 | 6 | 7.5 | 4.5 | 6.4 | 2.1 |
| πͺ Claude Opus 5 | $5.00 | $25.00 | $0.50 | $3.85 | 9 | 8.5 | 9 | 4 | 7.6 | 2.0 |
| πͺ Claude Fable 5.1 | $10.00 | $50.00 | $0.25 | $7.17 | 10 | 9.5 | 10 | 4.5 | 8.5 | 1.2 |
| π© GPT-6 Astra | $10.00 | $50.00 | $1.00 | $7.70 | 10 | 9.5 | 9.5 | 4 | 8.2 | 1.1 |
What "$/MTok" means: input, output and cache-hit rates are three different prices, so no single one lets you rank models. $/MTok collapses them into one dollar figure per million tokens using the average real-world mix β 7 : 2 : 1 cache hits : fresh input : output (0.7Γcache + 0.2Γinput + 0.1Γoutput). That is Artificial Analysis' standard ratio, and it reflects how production traffic actually looks: most tokens are re-read cached context, few are output.
Sanity check β this formula exactly reproduces the blended prices AA publishes for the five models it lists: Sonnet 5 $1.54, DeepSeek V4.1-Flash $0.18, GLM-5.3 $0.90, Opus 5 $3.85, Fable 5.1 $7.17.
All prices USD per 1M tokens, vendor list rates, 11 Sep 2026.
Scoring basis β all reproducible:
| Axis | Formula | Source |
|---|---|---|
| π§ Reasoning | (AA Intelligence Index β 30) Γ· 2.3 | Artificial Analysis v4.3, max effort |
| π» Coding | Terminal-Bench 4.0 % Γ· 6 | Official tbench.ai board, 10 Sep 2026; (est) where absent |
| π General | Judgement: HLE (no tools) + breadth of deployment + is it a general model at all | Mixed |
| β‘ Speed | 4 + 6Γ(tok/s β 50) Γ· 150, capped at 10 | AA measured output throughput |
| π― Capability | (reasoning + coding + general + speed) Γ· 4 | Average of the four axes |
| π° CP value | Capability Γ· $/MTok (7:2:1) | Derived from the above |
DeepSeek's 34.6 is 29Γ Fable 5.1's 1.2, but it buys you 6.4 capability against Fable's 8.5. If your task fails below a capability threshold, a high CP score is worthless: you pay twice and still don't ship. Read CP value only after you know which models clear your bar.
At 70% of the mix it drives $/MTok more than the headline rates do, and it reorders the table: Fable 5.1 ($7.17) is now cheaper than GPT-6 Astra ($7.70) purely on its $0.25 cache read vs Astra's $1.00 β despite identical $10/$50 list prices. DeepSeek reads cache at $0.006, 42Γ cheaper than Fable 5.1 and 167Γ cheaper than Astra. If your workload has no cache reuse, use the no-cache column in the pricing section instead.
Variant detail β every model at every setting
The scorecard uses one row per model family. In practice you buy a variant: an effort level, a Fast tier, a peak/off-peak window. This is where the real decisions get made β and where the biggest surprises are.
| Variant | Input | Output | Cache hit | $/MTok | Terminal-Bench 4.0 | ARC-AGI-2 | Notes |
|---|---|---|---|---|---|---|---|
π© GPT-6 Astra max | $10.00 | $50.00 | $1.00 | $7.70 | 58.2% | 95.0% | Benchmark ceiling, $3,267/run |
π© GPT-6 Astra xhigh | $10.00 | $50.00 | $1.00 | $7.70 | 57.9% | 93.3% | $2,351/run β 28% cheaper, same score |
π© GPT-6 Astra high | $10.00 | $50.00 | $1.00 | $7.70 | 57.9% | 92.1% | $2,269/run β the value setting |
π© GPT-6 Astra medium | $10.00 | $50.00 | $1.00 | $7.70 | 54.2% | 92.1% | $1,915/run |
π© GPT-6 Astra low | $10.00 | $50.00 | $1.00 | $7.70 | 50.6% | β | $1,557/run β still beats every non-Astra/Fable model |
πͺ Claude Fable 5.1 max | $10.00 | $50.00 | $0.25 | $7.17 | 57.9% | 90.0% | $6,244/run |
πͺ Claude Fable 5.1 xhigh | $10.00 | $50.00 | $0.25 | $7.17 | β | 90.0% | ARC cost/task $3.12 vs max's $4.49 |
πͺ Claude Fable 5.1 high | $10.00 | $50.00 | $0.25 | $7.17 | β | 88.8% | ARC cost/task $1.67 β 2.7Γ cheaper than max for β1.2 pts |
πͺ Claude Opus 5 max | $5.00 | $25.00 | $0.50 | $3.85 | 51.8% | 90.4% | $5,969/run |
πͺ Claude Opus 5 high (default) | $5.00 | $25.00 | $0.50 | $3.85 | β | 88.3% | ARC cost/task $1.45 vs max's $2.06; Arena coding Elo 1533 > max's 1525 |
| πͺ Claude Opus 5 fast mode | $10.00 | $50.00 | β | β | β | β | Research preview, 2.5Γ output tok/s, Opus 5 / 4.8 only |
πͺ Claude Sonnet 5 max | $2.00 | $10.00 | $0.20 | $1.54 | 12.4% | β | $9,604/run β the most expensive run on the whole board, for last place |
| πͺ Claude Haiku 4.5 | $1.00 | $5.00 | $0.10 | $0.77 | β | β | 200K ctx, no effort parameter |
π© GPT-5.6 Sol max | $4.00 | $20.00 | $0.40 | $3.08 | 37.3% | 92.5% | ARC-AGI-2 beats Opus 5 and Fable 5.1 |
π© GPT-5.6 Terra max | $2.00 | $12.00 | $0.20 | $1.74 | 21.5% | β | Mid tier |
π© GPT-5.6 Luna max | $0.20 | $1.20 | $0.02 | $0.17 | 17.3% | β | $347/run β cheapest run on the board |
π¨ GLM-5.3 max | $1.40 | $4.40 | $0.26 | $0.90 | 41.8% | β | Under Claude Code, not a native agent; $2,728/run |
| π¨ GLM-5.3-Flash | $0.15 | $0.50 | $0.03 | $0.10 | β | β | Arena coding Elo 1534 β above GLM-5.3-max's 1528 |
π¦ Grok 4.6 high <200K | $2.00 | $6.00 | $0.50 | $1.35 | 20.3% | 67.1% | Under xAI's own Grok Build harness |
π¦ Grok 4.6 high β₯200K | $4.00 | $12.00 | $1.00 | $2.70 | β | β | Whole request reprices at the threshold β 2Γ |
| π§ DeepSeek V4.1-Flash peak | $0.30 | $1.20 | $0.006 | $0.18 | 31.2%* | β | *Self-reported; absent from the official board |
| π§ DeepSeek V4.1-Flash off-peak | $0.15 | $0.60 | $0.003 | $0.09 | 31.2%* | β | 50% off outside 01β04 & 06β10 UTC MonβFri |
| π§ DeepSeek V4-Pro | $1.32 | $3.96 | $0.044 | $0.69 | ~27% (est) | 61.3% | Reroutes to V4.1-Flash 14 Sep |
| β¬ Composer 2.5 | $0.50 | $2.50 | $0.20 | $0.49 | ~15% (est) | β | 9.3 min/task; Next.js Evals 81% at $0.062/eval |
| β¬ Composer 2.5 Fast | $3.00 | $15.00 | $0.50 | $2.45 | β | β | 6.7 min/task (30% faster) for 5Γ the $/MTok |
Four things this table shows that the family-level scorecard hides:
- Effort is a bigger lever than model choice on Astra.
highscores 57.9% β identical tomax's 58.2% inside the CI β for 31% less run cost. And Astra atlow(50.6%) still beats every model here except Fable 5.1 and Opus 5 at max. Paying formaxon Astra buys essentially nothing. - Higher effort is not monotonic. Opus 5 at
highbeats Opus 5 atmaxon Arena coding Elo (1533 vs 1525), and GLM-5.3-Flash beats GLM-5.3-max there too (1534 vs 1528). Max effort is not a free upgrade β sweep it, don't assume it. - Composer 2.5 Fast is a bad trade on paper. 5Γ the token price for a 30% latency cut. Take it only when a developer is watching the cursor blink; never for batch work.
- Sonnet 5 at max is the worst value on the board. 12.4% on Terminal-Bench 4.0 for a $9,604 run β more than Fable 5.1 ($6,244) which scored 4.7Γ higher. Sonnet 5 is a fine general model; it is not an agentic-coding model, and max effort makes it expensive without making it good.
Pick one in 30 seconds
| πͺ Opus 5 β the default | πͺ Fable 5.1 β the ceiling | π§ DeepSeek V4.1-Flash β the value | |
|---|---|---|---|
| Advantage | 51 index at $3.85/MTok Β· TB-4.0 51.8% Β· Next.js Evals 94% Β· cache reads $0.50 | #1 index (53) Β· TB-4.0 57.9% Β· Next.js Evals 97% at $0.72/eval Β· cache reads $0.25 | $0.18/MTok β 39Γ cheaper than Fable Β· 194 tok/s Β· 0.96s TTFT Β· MIT weights Β· 384K max output |
| Don't use it ifβ¦ | β your evals at xhigh still fall shortβ you need >194 tok/sβ your budget is per-token not per-outcome | β you have a hard token budgetβ you need first token in <5sβ ZDR is required (400s without Anthropic authorisation) | β you need frontier agentic coding (31.2% TB-4.0)β you can't send data to a PRC-hosted APIβ you need vision-heavy enterprise SLAs |
| π© GPT-6 Astra | π¨ GLM-5.3 | π¦ Grok 4.6 | β¬ Composer 2.5 | |
|---|---|---|---|---|
| Advantage | TB-4.0 60% β best measured coding Β· 1.05M ctx Β· 27k output tok/task vs Fable's 78k | Best open-weights (45) Β· $0.90/MTok Β· 2.9s TTFT Β· $18/mo coding plan | $1.35/MTok Β· GDPval-AA 1663 (beats Astra's 1555) Β· 500K ctx | $0.49/MTok Β· ~200 tok/s class Β· $0.07/task on AA's Coding Agent Index |
| Don't use it ifβ¦ | β $7.70/MTok is too much β the priciest hereβ >272K prompts (2Γ input, 1.5Γ output)β you need published SWE-bench numbers | β you're a >$10B-revenue MaaS operator (licence review)β you need frontier reasoningβ you need a US/EU-hosted first-party API | β agentic coding matters (TB-4.0 20.3%)β you cross 200K tokens (whole request reprices 2Γ)β you need 1M context | β you need an API β there isn't oneβ you work outside Cursorβ you need any non-coding capability |
Anthropic's cache-read pricing. Fable 5.1 reads cache at 2.5% of input ($0.25/1M) vs Opus 5's 10% ($0.50/1M). Push cache reuse above the 70% the 7:2:1 ratio assumes and the 2Γ list-price gap between them keeps narrowing β Anthropic estimates ~25% lower cost for typical workloads and up to ~45% for highly agentic ones. Measure your own cache-hit rate before trusting any single-number price.
Price β the real numbers
All figures are vendor list price, USD per 1M tokens, as published on 11 Sep 2026.
Two mixes are shown. $/MTok (7:2:1) is the realistic one used everywhere else in this doc β cache hits dominate production traffic. No-cache (3:1) is the worst case: a cold prompt every call, nothing reused. The gap between the two columns is the value of prompt caching.
| Model | Input | Output | Cache hit | $/MTok (7:2:1) | No-cache (3:1) | Caching saves | Context | Max output |
|---|---|---|---|---|---|---|---|---|
| πͺ Claude Fable 5.1 | $10.00 | $50.00 | $0.25 | $7.17 | $20.00 | 64% | 1M | 128K |
| π© GPT-6 Astra | $10.00 | $50.00 | $1.00 | $7.70 | $20.00 | 62% | 1.05M | 128K |
| πͺ Claude Opus 5 | $5.00 | $25.00 | $0.50 | $3.85 | $10.00 | 62% | 1M | 128K |
| π© GPT-5.6 Sol | $4.00 | $20.00 | $0.40 | $3.08 | $8.00 | 62% | 1.05M | 128K |
| πͺ Claude Sonnet 5 | $2.00 | $10.00 | $0.20 | $1.54 | $4.00 | 62% | 1M | 128K |
| π¦ Grok 4.6 | $2.00 | $6.00 | $0.50 | $1.35 | $3.00 | 55% | 500K | not published |
| π¨ GLM-5.3 | $1.40 | $4.40 | $0.26 | $0.90 | $2.15 | 58% | 1M | 128K |
| πͺ Claude Haiku 4.5 | $1.00 | $5.00 | $0.10 | $0.77 | $2.00 | 62% | 200K | 64K |
| π§ DeepSeek V4-Pro (retiring 14 Sep) | $1.32 | $3.96 | $0.044 | $0.69 | $1.98 | 65% | 1M | 384K |
| β¬ Composer 2.5 | $0.50 | $2.50 | $0.20 | $0.49 | $1.00 | 51% | ~200K (unofficial) | not published |
| π§ DeepSeek V4.1-Flash | $0.30 | $1.20 | $0.006 | $0.18 | $0.53 | 66% | 1M | 384K |
| π© GPT-5.6 Luna | $0.20 | $1.20 | $0.02 | $0.17 | $0.45 | 62% | 1.05M | 128K |
| π¨ GLM-5.3-Flash | $0.15 | $0.50 | $0.03 | $0.10 | $0.24 | 58% | 1M | not published |
Four pricing traps that aren't in the headline rate:
| Trap | Model | Cost |
|---|---|---|
| Long-context surcharge | GPT-6 Astra | >272K input bills 2Γ input, 1.5Γ output |
| Threshold repricing | Grok 4.6 | β₯200K tokens reprices the entire request at $4/$12 |
| Cache write premium | GPT-6 Astra | Cache writes $12.50/1M β 25% above input |
| Off-peak discount | DeepSeek | 50% off outside 01:00β04:00 and 06:00β10:00 UTC MonβFri β $0.09/MTok |
Subscription alternatives (when per-token isn't how you buy):
| Plan | Price | What you get |
|---|---|---|
| GLM Coding Lite | $18/mo ($12.60 yearly) | 10,000 credits/week, GLM-5.3 + Flash |
| Cursor Pro | $20/mo | Composer 2.5 + all third-party models |
| GLM Coding Pro | $80/mo ($56 yearly) | 6Γ Lite quota, priority access |
| GLM Coding Max | $168/mo ($117.60 yearly) | 14Γ Lite quota, dedicated peak resources |
| Cursor Ultra | $200/mo | Highest allowance |
GLM off-peak usage costs 50% of the standard credit rate (off-peak = outside MonβFri 14:00β18:00 UTC+8). DeepSeek publishes no coding subscription.
Reasoning β Artificial Analysis Intelligence Index v4.3
The only index measuring all of these on one methodology. Max effort unless noted.
| Rank | Model | Index | vs leader | Cost per index point ($/MTok Γ· index) |
|---|---|---|---|---|
| 1= | πͺ Claude Fable 5.1 | 53 | β | $0.135 |
| 1= | π© GPT-6 Astra | 53 | β | $0.145 |
| 3 | πͺ Claude Opus 5 | 51 | β2 | $0.076 |
| 4 | π© GPT-5.6 Sol | 47 | β6 | $0.066 |
| 5 | π¨ GLM-5.3 | 45 | β8 | $0.020 |
| 6 | π¦ Grok 4.6 (high) | 44 | β9 | $0.031 |
| 7 | π§ DeepSeek V4.1-Flash | 40 | β13 | $0.0046 |
| 8 | πͺ Claude Sonnet 5 | 38 | β15 | $0.041 |
| 9 | π§ DeepSeek V4-Pro | 36 | β17 | $0.019 |
| β | β¬ Composer 2.5 | ~32 (est) | β21 | $0.015 |
The headline is the last column. DeepSeek V4.1-Flash buys an index point for 0.46 cents against Fable 5.1's 13.5 cents β a 29Γ efficiency gap. The frontier costs what it costs; the question is whether the last 13 index points change your outcome.
Reasoning depth, where vendors publish it:
| Benchmark | Fable 5.1 | Opus 5 | Astra | Grok 4.6 | DS V4.1-Flash | GLM-5.3 |
|---|---|---|---|---|---|---|
| Humanity's Last Exam (no tools) | 60.9% | 56.6% | 55% | 44% | 36.8% | not published |
| Humanity's Last Exam (with tools) | 65.0% | 63.6% | not published | not published | not published | 62.5% |
| GPQA Diamond | not published | not published | not published | not published | 90.9% | not published |
| SciCode | not published | not published | 56% | 53% | not published | not published |
| AA-Omniscience | not published | not published | 43 | 29 | not published | not published |
Anthropic, OpenAI, xAI and Z.ai publish none of them for their current flagships. Only DeepSeek still reports GPQA Diamond (90.9%). Any 2026 comparison built on SWE-bench Verified is comparing models on a benchmark their makers stopped reporting.
Coding β Terminal-Bench 4.0 (official leaderboard)
The full board from tbench.ai, updated 10 Sep 2026. 330 trials per row, resolution rate, with each run's total cost. This supersedes the vendor-quoted figures β it is one harness-declared board rather than nine vendor press releases.
| # | Model | Agent harness | Effort | Score (95% CI) | Run cost |
|---|---|---|---|---|---|
| 1 | π© GPT-6 Astra | Codex | max | 58.2% Β± 2.8 | $3,267 |
| 2 | πͺ Claude Fable 5.1 | Claude Code | max | 57.9% Β± 3.8 | $6,244 |
| 2 | π© GPT-6 Astra | Codex | xhigh | 57.9% Β± 2.7 | $2,351 |
| 2 | π© GPT-6 Astra | Codex | high | 57.9% Β± 3.0 | $2,269 |
| 5 | π© GPT-6 Astra | Codex | medium | 54.2% Β± 2.7 | $1,915 |
| 6 | πͺ Claude Opus 5 | Claude Code | max | 51.8% Β± 3.4 | $5,969 |
| 7 | π© GPT-6 Astra | Codex | low | 50.6% Β± 2.8 | $1,557 |
| 8 | πͺ Claude Fable 5 | Claude Code | max | 44.5% Β± 3.8 | $7,265 |
| 9 | π¨ GLM-5.3 | Claude Code | max | 41.8% Β± 3.2 | $2,728 |
| 10 | π© GPT-5.6 Sol | Codex | max | 37.3% Β± 3.8 | $2,542 |
| 11 | πͺ Claude Opus 4.8 | Claude Code | max | 23.6% Β± 3.6 | $6,481 |
| 12 | π© GPT-5.6 Terra | Codex | max | 21.5% Β± 3.3 | $1,734 |
| 13 | π¦ Grok 4.6 | Grok Build | high | 20.3% Β± 3.1 | $3,592 |
| 14 | Gemini 3.8 Flash | mini-SWE-agent | high | 19.1% Β± 3.4 | $1,829 |
| 15 | π© GPT-5.6 Luna | Codex | max | 17.3% Β± 2.8 | $347 |
| 16 | π¦ Grok 4.5 | Grok Build | high | 12.4% Β± 2.6 | $2,094 |
| 16 | πͺ Claude Sonnet 5 | Claude Code | max | 12.4% Β± 3.1 | $9,604 |
| 18 | Gemini 3.7 Flash | mini-SWE-agent | high | 11.2% Β± 2.4 | $1,262 |
Absent from the board: DeepSeek V4.1-Flash (self-reports 31.2%), DeepSeek V4-Pro, Cursor Composer 2.5, Gemini 3.1 Pro. Their rows in the scorecard stay estimated.
The harness is half the result. GLM-5.3's 41.8% is achieved under Claude Code, not a native Z.ai agent; the Gemini rows run under mini-SWE-agent; Grok runs under xAI's own Grok Build. A model's score here is a model + scaffold score, and swapping the scaffold moves it by more than most model upgrades do.
Three results worth staring at:
| Finding | Numbers |
|---|---|
| Astra's effort curve is nearly flat | high 57.9% at $2,269 vs max 58.2% at $3,267 β statistically identical for 31% less |
| GLM-5.3 is the open-weights story | 41.8% at $2,728 beats GPT-5.6 Sol's 37.3% at $2,542, and it is 4th among model families |
| Sonnet 5 at max is the board's worst value | 12.4% for $9,604 β the most expensive run listed, 4.7Γ below Fable 5.1 which cost $3,360 less |
The rest of the coding evidence
| Benchmark | Fable 5.1 | Opus 5 | Astra | Grok 4.6 | GLM-5.3 | DS V4.1-Flash | Composer 2.5 |
|---|---|---|---|---|---|---|---|
| CursorBench 3.2 | 73.4% | 70.0% | not pub. | 69.9% | not pub. | not pub. | 63.2% (v3.1) |
| Terminal-Bench 2.1 | not pub. | 89.1% | not pub. | 88.4% | 88.2% | 90.6% | 66% (v2, AA) |
| Terminal-Bench 3.0 | not pub. | not pub. | not pub. | 26% | 28.3% | 30.0% | not pub. |
| DeepSWE v1.1 | not pub. | not pub. | 73%* | 65.9% | 66.9% | 74.2% | not pub. |
| AutomationBench | 31.4% | 26.9% | 67%* | 67%* | 48.2%* | 54.8%* | not pub. |
| SWE-bench Multilingual | not pub. | not pub. | not pub. | not pub. | not pub. | not pub. | 79.8% |
| GDPval-AA v2 (Elo) | 1853 | 1824 | 1555* | 1663* | 1769 | not pub. | not pub. |
* Different harness/version β AutomationBench and GDPval-AA numbers come from AA, xAI and Z.ai runs that are not mutually comparable. Grok 4.6's GDPval-AA 1663 beating Astra's 1555 is AA's own measurement and is real; Z.ai's 1769 for GLM-5.3 is self-reported on a different run.
DeepSeek 90.6, Opus 5 89.1, Grok 4.6 88.4, GLM-5.3 88.2 β a 2.4-point spread that inverts on TB 4.0, where Opus 5 (51.8), GLM-5.3 (41.8) and Grok 4.6 (20.3) spread over 31 points. Vendors quoting TB 2.1 in Sep 2026 are quoting a solved benchmark.
Token efficiency is the sleeper coding metric. AA measured GPT-6 Astra at 27k output tokens per task vs Fable 5.1's 78k β 2.9Γ fewer. At identical $50/1M output pricing, that is a 2.9Γ real cost difference that the price table cannot show you.
Real-world and independent evals
Vendor benchmarks measure what vendors choose to measure. These are run by third parties on tasks that look like actual work.
Next.js Agent Evals β the most job-like benchmark here
Run by Vercel at nextjs.org/evals, last run 10 Sep 2026. 27 evals from vercel/next-evals-oss: each is a prompt plus a real Next.js app, the agent works in an isolated Vercel Sandbox, and withheld assertions run afterwards. Scored pass@4. Crucially it publishes cost and wall-clock per eval, and it is the only current board that includes Cursor Composer 2.5.
| # | Model | Agent | Avg duration | Avg cost/eval | Pass | +AGENTS.md |
|---|---|---|---|---|---|---|
| 1 | πͺ Claude Fable 5.1 (high) | Claude Code | 247.6s | $0.722 | 97% | 97% |
| 2 | πͺ Claude Opus 5 | Claude Code | 368.4s | $1.96 | 94% | 97% |
| 3 | π© GPT-6 Astra (high) | Codex | 251.9s | $0.891 | 90% | 97% |
| 4 | Kimi K3 | OpenCode | 352.3s | $0.281 | 84% | 97% |
| 5 | π¨ GLM 5.2 | OpenCode | 395.5s | $0.152 | 81% | 97% |
| 6 | πͺ Claude Sonnet 5 | Claude Code | 263.0s | $0.386 | 81% | 97% |
| 7 | β¬ Composer 2.5 | Cursor | 273.6s | $0.062 | 81% | 94% |
| 8 | MiniMax M3 | OpenCode | 260.1s | $0.053 | 81% | 90% |
| 9 | Kimi K2.7 Code | OpenCode | 318.4s | $0.139 | 74% | 90% |
Superseded rows (not re-run): GPT-5.6 Sol (ultra) 77% Β· Claude Fable 5 (high) 77% Β· Grok 4.6 71% Β· Claude Opus 4.8 68% Β· Gemini 3.1 Pro 58% Β· Composer 2.0 58%. GLM-5.3 and both DeepSeek V4.x models have never been run β not found.
This board reorders everything. Sonnet 5 scores 81% here against 12.4% on Terminal-Bench 4.0 β and Composer 2.5 matches it at 1/6 the cost ($0.062 vs $0.386 per eval), while Fable 5.1 wins outright at $0.722, a third of Opus 5's $1.96. On ordinary framework work the cheap models are much closer to the frontier than the agentic benchmarks suggest.
Handing the agent bundled Next.js docs lifts Composer 2.5 from 81% β 94% and Sonnet 5 from 81% β 97% β matching Fable 5.1's unaided score, for the price of a context file. Six of the nine current models converge on 97% once documentation is supplied. Before paying for a better model, give the one you have better docs.
LiveBench β contamination-resistant, monthly refresh
livebench.ai, release LiveBench-2026-06-25, 52 models, with cost per task.
| # | Model | Overall | Reasoning | Coding | Agentic coding | Cost/task |
|---|---|---|---|---|---|---|
| 1 | πͺ Claude Fable 5.1 (max) | 83.4 | 91.7 | 86.4 | 66.1 | $1.212 |
| 2 | πͺ Claude Fable 5 (max) | 83.0 | 89.7 | 86.0 | 62.2 | $1.439 |
| 3 | π© GPT-6 Astra (max) | 82.2 | 92.7 | 80.4 | 57.3 | $0.736 |
| 4 | Muse Spark 1.3 (xhigh) | 81.6 | 89.7 | 81.1 | 64.1 | $0.219 |
| 5 | π§ DeepSeek V4.1-Flash (max) | 81.1 | 86.7 | 80.0 | 77.3 | $0.029 |
| 6 | π© GPT-5.6 Sol (max) | 81.0 | 91.7 | 83.9 | 56.2 | $0.515 |
| 8 | πͺ Claude Opus 5 (max) | 80.1 | 91.2 | 81.4 | 65.2 | $0.699 |
| 12 | π¦ Grok 4.6 | 78.0 | 90.5 | 76.8 | 57.0 | $0.207 |
| 17 | π§ DeepSeek V4-Pro 0813 | 77.4 | β | β | β | β |
| 18 | Gemini 3.1 Pro (high) | 77.0 | β | β | 44.1 | β |
| 23 | π¨ GLM-5.3 | 76.1 | β | β | β | β |
| 24 | πͺ Claude Sonnet 5 (xhigh) | 76.0 | β | β | β | β |
DeepSeek V4.1-Flash scores 77.3 on agentic coding β the highest in that column, above Fable 5.1's 66.1 and Opus 5's 65.2, at $0.029/task versus $1.212 (42Γ cheaper). That directly contradicts its 31.2% self-reported Terminal-Bench 4.0 and its absence from the official board. One of those two measurements is misleading and I cannot tell you which β run your own eval before betting either way. Cursor Composer 2.5 is not on LiveBench.
ARC-AGI-2 β abstract reasoning, with cost per task
arcprize.org, semi-private eval, generated 4 Sep 2026. Human panel baseline = 100% at $17/task.
| Model (effort) | Score | Cost/task |
|---|---|---|
| π© GPT-6 Astra (max) | 95.0% | $1.12 |
| π© GPT-6 Astra (xhigh) | 93.3% | $0.83 |
| π© GPT-5.6 Sol (max) | 92.5% | $1.44 |
| π© GPT-6 Astra (high / medium) | 92.1% | $0.67 / $0.48 |
| πͺ Claude Opus 5 (max) | 90.4% | $2.06 |
| πͺ Claude Fable 5.1 (max / xhigh) | 90.0% | $4.49 / $3.12 |
| πͺ Claude Fable 5.1 (high) | 88.8% | $1.67 |
| πͺ Claude Opus 5 (high) | 88.3% | $1.45 |
| Gemini 3.1 Pro (preview) | 77.1% | $0.96 |
| π¦ Grok 4.6 (xhigh) | 67.1% | $0.76 |
| π§ DeepSeek V4 Flash 0731 (max) | 61.4% | $0.04 |
| π§ DeepSeek V4-Pro 0813 (max) | 61.3% | $0.60 |
| π¨ GLM-5.2 | 22.8% | $0.25 |
OpenAI owns this benchmark. Astra at medium (92.1%, $0.48/task) beats Opus 5 at max (90.4%, $2.06) and Fable 5.1 at max (90.0%, $4.49) β 4.3Γ and 9.4Γ cheaper respectively. If your workload looks like abstract pattern reasoning, this is the clearest model-choice signal in the whole document. GLM-5.3, Sonnet 5 and Composer 2.5 are unmeasured.
GPT-6 Astra scores 99.95% under ARC's "Provider Adapter" harness and 62.71% under the Standard harness β same model, same effort. Anyone quoting "99.9% on ARC-AGI-3" is quoting the scaffold, not the model. Claude Opus 5 (high) scores 30.16% Standard; Fable 5.1, Sonnet 5, GLM-5.3, DeepSeek and Composer are all unmeasured.
Human preference β Arena Elo
LMArena has moved to arena.ai. Pairwise human votes; Β±CI shown.
| Board | Snapshot | Top of the board |
|---|---|---|
| Overall text | 3 Sep 2026, 8.0M votes | Fable 5 1507, Opus 4.6-high 1505, Fable 5.1-max 1504, Opus 4.7-high 1502 β¦ Opus 5-high 1493 (#9), Opus 5-max 1488, Gemini 3.1 Pro 1487, Sol-xhigh 1483, GLM-5.3-max 1482, GLM-5.3-Flash 1474, Sonnet 5-high 1462, Grok 4.6-high 1461, DeepSeek V4-Pro 1458 |
| Coding | 2 Sep 2026, 1.68M votes | Opus 4.7-high 1552, Opus 4.6-high 1552, Fable 5 1551, Kimi K3-max 1542 β¦ GLM-5.3-Flash 1534, Opus 5-high 1533, Sol-xhigh 1529, GLM-5.3-max 1528, Opus 5-max 1525 |
| WebDev Arena | 8 Sep 2026, 663k votes | Astra-max 1796, Fable 5.1-max 1764, Opus 5-max 1688, Qwen3.8-max 1685, Kimi K3-max 1674 β¦ Grok 4.6-high 1624, Sol-xhigh 1617, GLM-5.3-max 1613, DeepSeek V4.1-Flash 1620, Sonnet 5-high 1538 (#30) |
Humans and benchmarks disagree sharply on the overall text board β the top four are older Anthropic models (Fable 5, Opus 4.6, Opus 4.7), with Opus 5 only 9th. Vote counts explain part of it: Fable 5.1-max has 2,906 votes against Opus 4.6-high's 72,099, so the new models' intervals are 2β3Γ wider. Treat the text board as lagging; WebDev Arena, with a clear Astra β Fable 5.1 β Opus 5 ordering, tracks the benchmarks far better. GPT-6 Astra and Composer 2.5 are entirely absent from the text and coding boards.
Real usage β what people actually run
Adoption is not quality, but it is the one number no vendor can spin.
| OpenRouter, week to 10 Sep 2026 | Tokens | Vercel AI Gateway, 9β10 Sep 2026 | Team share | |
|---|---|---|---|---|
| Hy4 preview (Tencent) | 19.1T | GLM-5.3-Flash | 35.8% | |
| GPT-5.6 Luna | 14.2T | Grok 4.6 | 11.9% | |
| DeepSeek V4 Flash 0731 | 12.4T | Claude Haiku 4.5 | 7.7% | |
| GLM-5.3-Flash | 12.3T | Gemini 2.5 Flash | 6.6% | |
| GLM-5.3 | 3.06T | Claude Sonnet 5 | 6.3% | |
| Claude Opus 5 | 1.72T (#15) | GPT-5.6 Luna | 6.1% | |
| GPT-5.6 Sol | 1.71T (#16) | MiniMax M3 | 6.0% | |
| Claude Sonnet 5 | 1.37T (#19) |
Real-world usage is dominated by cheap models, by an order of magnitude. GLM-5.3-Flash and DeepSeek Flash together move ~25T tokens/week on OpenRouter while Opus 5 moves 1.72T; on Vercel's gateway a single cheap model holds 35.8% of teams. The frontier wins benchmarks; the floor wins deployments. (OpenRouter's % column is week-over-week change, not share β and it excludes first-party API traffic, where Anthropic and OpenAI are strongest, so this understates them.)
METR time horizons β stale, and worth knowing why
METR measures the task length a model completes with 50% reliability β the best single proxy for long-horizon agentic capability. Last updated 8 May 2026, so not one model in this comparison is measured. The frontier as METR last saw it: Claude Mythos Preview 17.4h, Claude Opus 4.6 12.0h, Gemini 3.1 Pro 6.4h, GPT-5.4 5.7h, Claude Opus 4.5 4.9h. METR's fitted doubling time is 128.7 days (CI 104β158). Extrapolating that to Opus 5 or Fable 5.1 would be guesswork, so this doc doesn't. METR also flags that "measurements above 16 hrs are unreliable with our current task suite".
What the official SWE-bench boards do and don't show
swebench.com's newest entry is 26 Feb 2026 β no model in this comparison appears. Its apples-to-apples subset (mini-SWE-agent 2.0.0, single attempt) tops out at Claude Opus 4.5 76.8% for $376.95 per run. Scale's SWE-bench Pro is also behind: Muse Spark 1.1 61.5%, GPT-5.4 xhigh 59.1%, Claude Opus 4.6 51.9%. The widely-quoted "Fable 5.1 leads SWE-bench Pro at 81.2%" is not on Scale's official board β it appears only on aggregator sites and is excluded here.
Speed
| Model | Output tok/s | Time to first token | Interactive feel |
|---|---|---|---|
| π§ DeepSeek V4.1-Flash | 194.3 | 0.96s | Instant |
| β¬ Composer 2.5 | ~200 (est) | not published | Instant (Cursor's whole pitch) |
| πͺ Claude Sonnet 5 | 73.2 | 212.72s | Long wait, then fast |
| πͺ Claude Fable 5.1 | 66.3 | 304.16s | Longest wait |
| π© GPT-5.6 Sol | 59.6 | 163.50s | Long wait |
| π¨ GLM-5.3 | 59.4 | 2.91s | Responsive |
| π© GPT-6 Astra | 54.4 | 339.78s | Longest wait |
| π¦ Grok 4.6 | 53.3 | 41.39s | Noticeable pause |
| πͺ Claude Opus 5 | 51.7 | 107.85s | Long wait |
These are max-effort measurements where TTFT includes the entire thinking phase. Models that stream their reasoning (GLM-5.3 at 2.91s, DeepSeek at 0.96s) look 100Γ faster than models that hide it (Claude defaults to display: "omitted", OpenAI likewise) β but the hidden-reasoning models are thinking, not stalling. Drop to low/medium effort and Claude's TTFT falls to single-digit seconds. Throughput (tok/s) is the honest cross-vendor number; TTFT is an artefact of effort setting and reasoning-display policy.
That is why the Speed score above uses throughput only.
Two speed levers the table doesn't capture:
- Claude fast mode (Opus 5 / Opus 4.8, research preview): up to 2.5Γ output tok/s at $10/$50 β same price as Fable 5.1, at Opus capability.
- Composer 2.5 Fast: 6.7 min/task vs 9.3 min standard (~30% faster) for 6Γ the list rates ($3/$15 vs $0.50/$2.50) β 5Γ on $/MTok.
Estimated numbers β and the scoreboard on how they did
Where no direct benchmark existed, numbers were bridged through models appearing on both benchmarks. The official Terminal-Bench 4.0 board has since falsified two of them, which is the most useful thing in this section: it tells you how much to trust the ones still outstanding.
Estimates that have now been checked against real data
| Estimate | I said | Official board says | Error | Verdict |
|---|---|---|---|---|
| GLM-5.3 TB 4.0 | ~29% | 41.8% | +12.8 pts | β Wrong β under-rated by 31%. The TB 3.0 β 4.0 bridge ratio (1.04) did not transfer. |
| Claude Sonnet 5 TB 4.0 | ~30% | 12.4% | β17.6 pts | β Badly wrong β over-rated by 2.4Γ. My "Claude models over-perform their index on terminal tasks" correction was exactly backwards for Sonnet. |
| Grok 4.6 TB 4.0 | 17% (via AA) | 20.3% | +3.3 pts | β Close β within the harness-variation noise. |
Two of three estimates missed by more than the benchmark's own spread between adjacent models. Both scorecard rows have been corrected to the measured values, and GLM-5.3's CP value rose from 6.4 to 6.9 as a result. Do not use bridged estimates to choose a model when a real measurement exists.
Estimates still outstanding
| Estimate | Bridge | Working | Confidence |
|---|---|---|---|
| Composer 2.5 TB 4.0 β 15% | TB 2.0 = 69.3 (Cursor) / TB v2 = 66% (AA) | TB 2.1β3.0 collapse ratio ~0.31 across GLM, DeepSeek and Grok, extrapolated below its measured range | π¨ Very low β and now doubly suspect, since the same method under-rated GLM-5.3 by 31%. Range 8β30%. Composer scores 81% on Next.js Evals, so treat this figure as measuring frontier-agentic difficulty, not everyday competence. |
| DeepSeek V4-Pro TB 4.0 β 27% | TB 2.1 = 87.9 vs V4.1-Flash 90.6 | Below V4.1-Flash on both available axes | β οΈ Low. Moot after 14 Sep β the model routes to V4.1-Flash. |
| Composer 2.5 AA index β 32 | Not on AA's index (no public API to test) | AA Coding Agent Index 62 at launch (3rd, May 2026); the Intelligence Index weights non-coding reasoning it was never trained for | π¨ Very low. Range 25β38. |
| Composer 2.5 ~200 tok/s | Composer 2 measured ~200 tok/s, Composer 1 ~250 | Cursor published no tok/s for 2.5 | β οΈ Low. Carried forward a generation. |
| DeepSeek V4.1-Flash TB 4.0 = 31.2% | Not an estimate β DeepSeek's own figure | Absent from the official board | β οΈ Vendor-reported, unverified. LiveBench scores its agentic coding highest of all models (77.3), which is hard to square with 31.2%. |
What is not estimated: every price, context window and release date, plus every Terminal-Bench 4.0, Next.js Evals, LiveBench, ARC-AGI and Arena figure in this document.
How to choose
Hard blockers β these decide it outright
| If⦠| Then |
|---|---|
| You need frontier agentic coding, price secondary | GPT-6 Astra (TB 4.0 58.2%) or Fable 5.1 (57.9%) β a tie; pick on price |
| Your per-token bill is the constraint | DeepSeek V4.1-Flash β 29Γ better index-per-dollar |
| You must self-host or own the weights | DeepSeek V4.1-Flash (MIT) β GLM-5.3's licence has a revenue clause |
| You cannot send data to a PRC-hosted API | Rules out DeepSeek and GLM entirely |
| You need Zero Data Retention | Rules out Fable 5.1 (400s without Anthropic authorisation) |
| You work inside Cursor and want cheap fast edits | Composer 2.5 β it is genuinely good at this and nothing else |
| You need >500K context | Rules out Grok 4.6 |
| You need an API | Rules out Composer 2.5 β there isn't one |
| You need published SWE-bench Verified numbers | Only DeepSeek still publishes comparable classics |
Everyone else β weighted quiz
Start at 0. Add points:
| Answer yes β | Points |
|---|---|
| Long-horizon agentic runs (>30 min autonomous) | +3 Fable 5.1 / +2 Astra |
| Heavily cached agent loop (same prefix every turn) | +3 Fable 5.1 (cache reads $0.25 β half of Opus 5's) |
| Output-heavy, low cache reuse | +2 Opus 5, +2 Astra (27k tok/task vs Fable's 78k) |
| High request volume, moderate difficulty | +3 DeepSeek V4.1-Flash / +2 GLM-5.3 |
| Interactive UI where users watch tokens appear | +3 DeepSeek / +2 GLM-5.3 |
| Non-coding knowledge work (docs, research, slides) | +2 Fable 5.1 / +2 Astra |
| Fixed monthly budget, not per-token | +3 GLM Coding Plan ($18/mo) |
| Enterprise procurement, US/EU hosting, SLAs | +2 Claude / +2 OpenAI, β3 DeepSeek/GLM |
Take Claude Opus 5. It is within 0.9 capability points of the best at half the price, it is the only model here whose vendor publishes first-party Terminal-Bench 4.0, and Anthropic's own guidance is to escalate to Fable 5.1 only when your evals on Opus at higher effort measurably fall short.
Pros
| Dimension | πͺ Claude (Fable 5.1 / Opus 5 / Sonnet 5) | π© OpenAI (Astra / Sol) | π¦ Grok 4.6 | π¨ GLM-5.3 | π§ DeepSeek V4.1-Flash | β¬ Composer 2.5 |
|---|---|---|---|---|---|---|
| Price | Sonnet 5 at $1.54/MTok is the cheapest credible frontier-adjacent option | β | β $1.35/MTok for index 44 | β $0.90/MTok for index 45 β best open-weights value | β $0.18/MTok, $0.09 off-peak | β $0.49/MTok, $0.07/task |
| Reasoning | β Index 53 (#1/200), HLE 60.9% | β Index 53, AA-Omniscience 43 | GDPval-AA 1663 beats Astra | β #1 open-weights at 45 | GPQA Diamond 90.9% | β |
| Coding | β TB 4.0 57.9 / 51.8 Β· Next.js Evals 97% / 94% β 1st and 2nd | β TB 4.0 58.2%, board leader Β· ARC-AGI-2 95% | β | β TB 4.0 41.8% β 4th family, best open-weights | β LiveBench agentic coding 77.3 β highest of any model | β Next.js Evals 81% at $0.062/eval, 94% with AGENTS.md |
| Speed | Fast mode: 2.5Γ output on Opus 5 | β | β | β 2.91s TTFT | β 194 tok/s, 0.96s TTFT | β ~200 tok/s class |
| Context | β 1M / 128K out; 300K out via batch beta | β 1.05M β largest | 500K | β 1M | β 1M ctx, 384K max output β largest output | ~200K |
| Caching | β $0.25/1M cache reads on Fable 5.1 (2.5% of input) | β | $0.50/1M | $0.26/1M | β $0.006/1M β 42Γ cheaper than Fable | $0.20/1M |
| Openness | β | β | β | β Open weights | β MIT, no revenue clause | β |
Cons
| Dimension | πͺ Claude | π© OpenAI | π¦ Grok 4.6 | π¨ GLM-5.3 | π§ DeepSeek | β¬ Composer 2.5 |
|---|---|---|---|---|---|---|
| Price | β Fable 5.1 at $7.17/MTok is second-priciest | β Astra $7.70/MTok β priciest here; cache writes $12.50/1M; 2Γ over 272K | β | β | β | β Fast variant is 6Γ standard ($3/$15) |
| Reasoning | Sonnet 5 at index 38 is #42/200 | β | β β9 vs leader | β β8 vs leader | β β13 vs leader | β Not on any reasoning index |
| Coding | β Sonnet 5: TB 4.0 12.4% for a $9,604 run β worst value on the board | β Publishes no first-party benchmark numbers at all | β TB 4.0 20.3% β bottom third | β Scored under Claude Code, not a native agent | β Self-reports TB 4.0 31.2% but is absent from the official board | β Not on TB 4.0, LiveBench, Arena or ARC β 16 weeks stale |
| Speed | β 52β73 tok/s β bottom half | β 54 tok/s, slowest TTFT (339s) | β 53 tok/s | β 59 tok/s | β | β No published tok/s for 2.5 |
| Context | β | β Surcharge past 272K | β 500K, and β₯200K reprices the whole request | β | β | β ~200K, unofficial; ~70β120K effective per third parties |
| Availability | β Fable 5.1 blocked under ZDR; no Priority Tier on Opus 5 / Sonnet 5 / Fable 5.1 | β No GPT-6 mini tier | β Max output tokens not documented | β Licence review for >$10B MaaS operators | β V4-Pro silently reroutes 14 Sep | β No API. Cursor only. |
| Transparency | β | β Named 8 benchmarks at launch, published zero scores | β Quotes a superseded index version (61 on v4.2 vs 44 on v4.3) | β Active params not disclosed | β | β No official context window, no tok/s |
Similarities β stop weighing these
- All eight are reasoning models with adjustable effort/thinking. None is a pure completion model.
- 1M context is table stakes β Claude, GPT, GLM and DeepSeek all ship it. Only Grok (500K) and Composer (~200K) don't.
- 128K max output on Claude, GPT and GLM. DeepSeek's 384K is the only outlier.
- All support vision except DeepSeek V4-Pro (V4.1-Flash added it).
- Prompt caching everywhere, at 0.6%β10% of input price.
- Nobody publishes SWE-bench Verified for their current flagship except DeepSeek's older cards.
- All frontier models now measure on agentic benchmarks (Terminal-Bench, GDPval, AutomationBench) rather than the 2024β25 knowledge tests.
Every model here is buying the same thing β agentic task completion β and the only real question is what you pay per completed task. Claude and OpenAI charge ~20β40Γ DeepSeek's rate for the top ~13 index points and ~25 points of Terminal-Bench 4.0. If your tasks fail without those points, the premium is cheap; if they don't, it's pure waste.
Trend β how the frontier moved in 2026
| Date | Event | Index at the time |
|---|---|---|
| 18 May 2026 | Cursor Composer 2.5 β 3rd on AA Coding Agent Index at 1/10 to 1/60 the cost per task | Coding Agent Index 62 |
| 30 Jun 2026 | Claude Sonnet 5 | 38 |
| 9 Jul 2026 | GPT-5.6 family (Sol / Terra / Luna) | 47 / 42 / 38 |
| 24 Jul 2026 | Claude Opus 5 β ARC-AGI 3 at 3Γ the next-best model | 51 |
| 12 Aug 2026 | Grok 4.6 | 44 |
| 14 Aug 2026 | GLM-5.3 β open weights take #1 open slot at 45 | 45 |
| 1 Sep 2026 | Claude Fable 5.1 β cache reads cut 75% to $0.25/1M | 53 π |
| 3 Sep 2026 | GPT-6 Astra β ties Fable 5.1 at 53, wins TB 4.0 at 60% | 53 π |
| 10 Sep 2026 | DeepSeek V4.1-Flash β 40 at $0.18/MTok, 194 tok/s | 40 |
| 14 Sep 2026 | DeepSeek V4-Pro reroutes to V4.1-Flash | β |
Three things the curve shows:
- The top is a two-horse tie and it changes every fortnight. Fable 5.1 (1 Sep) and Astra (3 Sep) both landed on 53 within 48 hours. Neither lead is durable enough to architect around β build model-agnostic.
- The open-weights gap has closed to 8 points and the price gap has not. GLM-5.3 at 45 costs 9Γ less than the 53s. In Jan 2026 that gap was far wider on both axes.
- The floor is rising much faster than the premium is falling. DeepSeek V4.1-Flash buys index 40 at 0.46Β’/point, while everything at index 47+ still costs 6.6β14.5Β’/point. Each new cheap model lifts what you get for near-nothing; the frontier's price per point has not moved.
Links
| Claude | OpenAI | xAI | Z.ai | DeepSeek | Cursor | |
|---|---|---|---|---|---|---|
| Docs | platform.claude.com | developers.openai.com | docs.x.ai | docs.z.ai | api-docs.deepseek.com | cursor.com/docs |
| Pricing | Pricing | Pricing | Pricing | Pricing | Pricing | Pricing |
| Announcement | Fable 5.1 Β· Opus 5 | GPT-6 Astra | Grok 4.6 | GLM-5.3 | V4.1-Flash | Composer 2.5 |
| Weights | β | β | β | HF (bespoke licence) | HF (MIT) | β |
| Subscription | claude.ai | β | β | z.ai/subscribe | β | cursor.com/pricing |
Every one of them moved in the fortnight before publication.
| Board | What it measures | Freshness |
|---|---|---|
| Terminal-Bench 4.0 | Agentic coding, with harness + run cost | β 10 Sep 2026 |
| Next.js Agent Evals | Real framework work, cost + duration per eval | β 10 Sep 2026 |
| LiveBench | Contamination-resistant, incl. agentic coding | β Jun 2026 release |
| ARC-AGI | Abstract reasoning, cost per task | β 4 Sep 2026 |
| Artificial Analysis | One index across all vendors, tok/s, TTFT | β continuous |
| Arena Β· coding Β· WebDev | Human preference Elo | β 2β8 Sep 2026 |
| OpenRouter rankings Β· Vercel AI Gateway | Real-world adoption | β 10 Sep 2026 |
| METR time horizons | Long-horizon agentic capability | β 8 May 2026 β stale |
| SWE-bench Β· SWE-bench Pro | Classic SWE tasks | β Feb 2026 β stale |
| Aider polyglot | Polyglot editing | β Nov 2025 β abandoned |
Gemini 3.1 Pro lists at $2.00 / $12.00 per 1M (rising to $4/$18 above 200K), and Gemini 3.8 Flash launched 2 Sep 2026 at $0.75 / $3.75 introductory β moving to $1.50 / $7.50 on 1 Jan 2027. It sits between Sonnet 5 and GLM-5.3 on price. (Third-party pricing aggregators, not verified against Google's own pricing page β treat as indicative.)
Background
Skip unless you're writing an ADR.
| Vendor | First model | Business model | |
|---|---|---|---|
| Claude | Anthropic | Claude 1, 2023 | First-party API + Bedrock / Vertex / Foundry / AWS; claude.ai subscriptions |
| GPT | OpenAI | GPT-3, 2020 | First-party API + Azure; ChatGPT subscriptions |
| Grok | xAI | Grok-1, 2023 | First-party API; X Premium bundling |
| GLM | Z.ai (Zhipu AI) | GLM-130B, 2022 | Open weights + hosted API + $18β168/mo coding plans |
| DeepSeek | DeepSeek | DeepSeek LLM, 2023 | MIT open weights + very low-cost hosted API |
| Composer | Cursor (Anysphere) | Composer 1, Oct 2025 | Not sold separately β bundled into Cursor's $20β200/mo plans |
Architecture, where disclosed: DeepSeek V4.1-Flash is 552B backbone with 8B active on prefill / 16B on decode (763B total file size); DeepSeek V4-Pro was 1.57T total / 48B active. GLM-5.3 is 753B total, active params undisclosed. Composer 2.5 is reported as a 1.04T-total / 32B-active MoE built on a Moonshot Kimi K2.5 checkpoint (third-party reporting, not confirmed by Cursor). Anthropic, OpenAI and xAI disclose nothing.
Composer lineage: Composer 1 (Oct 2025, CursorBench 38.0) β Composer 1.5 β Composer 2 (19 Mar 2026, 61.3, and a 7Γ price cut to $0.50/$2.50) β Composer 2.5 (18 May 2026, 63.2 on the harder v3.1 set). The jump from 1 to 2 came from Cursor's first continued-pretraining run; 2.5 added targeted textual feedback in RL and 25Γ more synthetic tasks grounded in real codebases.
Caveats
- Two of my three bridged estimates were wrong, and the section above says by how much. GLM-5.3 was under-rated by 12.8 points, Sonnet 5 over-rated by 17.6. Both are now corrected to measured values. The remaining estimates β all Composer 2.5 and DeepSeek V4-Pro β deserve less trust than their stated ranges imply.
- Β±3 points is the noise floor even on measured rows. The official Terminal-Bench 4.0 CIs run Β±2.7 to Β±3.8, so Astra's 58.2% and Fable 5.1's 57.9% are a tie, not a ranking. Anthropic and Artificial Analysis also disagree by 3β4 points running the same model on the same benchmark.
- The harness is part of every agentic score. GLM-5.3's 41.8% is under Claude Code; Grok runs under Grok Build; Gemini under mini-SWE-agent. ARC-AGI-3 shows the extreme case β GPT-6 Astra scores 99.95% vs 62.71% on the same model at the same effort, purely on scaffold. Never compare two agentic scores without checking the harness.
- xAI's own Grok 4.6 page claims index 61. That is AA's v4.2 index (before Terminal-Bench 4.0 replaced ΟΒ³-Banking on 7 Sep). On the current v4.3 index Grok 4.6 scores 44. Both are "correct"; they are different rulers. Every index number in this doc is v4.3.
- OpenAI's own benchmark claims remain unverified β openai.com returned 403 to every automated fetch. Every Astra number here comes from an independent board (tbench.ai, arcprize.org, nextjs.org, livebench.ai), not from OpenAI. The viral "99.9% on ARC-AGI-3" is the Provider Adapter harness, not the model; OpenAI's claimed 96% GPQA Diamond and 74.1% DeepSWE appear only on third-party blogs and are excluded.
- Grok 4.6's release date is disputed β x.ai and AA's article say 12 Aug 2026; AA's changelog says 21 Jul 2026. Vendor date used.
- Composer 2.5's context window is unofficial. Cursor publishes no figure. 200K comes from third-party analysis; one source puts the effective window after truncation at 70β120K.
- TTFT is not cross-vendor comparable β see the warning in the Speed section. It is reported for completeness, excluded from scoring.
- CP value divides by price, so it always favours cheap models β it answers "what do I get per dollar", never "is this good enough". The capability score weights its four axes equally, which flatters fast models. Both formulas are given so you can reweight.
- The official SWE-bench boards are too stale to use. swebench.com's newest entry is 26 Feb 2026 and Scale's SWE-bench Pro is not much newer β no model in this comparison appears on either. The circulating "Fable 5.1 leads SWE-bench Pro at 81.2%" is not on Scale's board; it is excluded.
- DeepSeek V4.1-Flash's two coding measurements flatly contradict each other. LiveBench puts its agentic coding highest of all models (77.3); DeepSeek self-reports 31.2% on Terminal-Bench 4.0 and is absent from the official board. I have not resolved this β run your own eval before relying on either.
- METR is 4 months stale (8 May 2026) and measures none of these models. Its 128.7-day doubling time is real; extrapolating it to Opus 5 or Fable 5.1 would not be, so this doc doesn't.
- Arena's overall text board disagrees with every benchmark β its top four are older Anthropic models and Opus 5 sits 9th. New models there have 2,900β35,000 votes against the leaders' 72,000+, so their intervals are much wider. Treat it as lagging.
Sources
Claude models overview Β· Claude Fable 5.1 launch Β· Claude Opus 5 launch Β· OpenAI pricing Β· OpenAI models Β· GPT-6 Astra system card Β· xAI pricing Β· Grok 4.6 Β· GLM-5.3 blog Β· Z.ai pricing Β· GLM Coding Plan Β· DeepSeek pricing Β· DeepSeek V4.1-Flash card Β· Composer 2.5 Β· Cursor models Β· AA leaderboard Β· AA: benchmarking GPT-6 Astra Β· AA: Composer 2.5 Coding Agent Index Β· Terminal-Bench 4.0 leaderboard Β· Next.js Agent Evals Β· next-evals-oss Β· LiveBench Β· ARC Prize leaderboard Β· Arena text Β· Arena coding Β· WebDev Arena Β· Design Arena Β· OpenRouter rankings Β· Vercel AI Gateway Β· METR time horizons Β· SWE-bench Β· SWE-bench Pro Β· APIs and pages queried 11 Sep 2026.
Disclaimer: These documents are personal notes. The information may be incomplete, out of date or wrong, so check it before you rely on it.