Measured performance
Benchmarks
Pick the work, and the models measured best at it come first. Every figure comes from the marketplace's own runs, with what each answer costs and how long it takes.
Pick the work, and the models measured best at it come first. Every figure comes from the marketplace's own runs, with what each answer costs and how long it takes.
Models with at least 50% of the weight measured are ranked by match first. The rest follow without one, marked limited evidence, the unmeasured last. Any column re-sorts the board.
| # | Model | ||||||
|---|---|---|---|---|---|---|---|
| 1 | DeepSeek V4.1 FlashConsensus Protocol | 91 | 100%Claude | 84.0% | 81.9%mean of 3 suites | $0.300$0.200 in · $0.600 out | 35 tok/s |
| 2 | DeepSeek V4.1 Flash 1017 UncensoredDealignAI · Consensus Protocol | 90 | 100%Claude | 76.0% | 79.6%mean of 3 suites | $0.300$0.200 in · $0.600 out | 36 tok/s |
| 3 | Qwen3.8 Flash Next AbliteratedDealignAI · ImpCC | 78 | 86.1%Empryo | 77.6% | 60.8%mean of 3 suites | $0.230$0.150 in · $0.470 out | 65 tok/s |
| 4 | Ling 3.0 Flash2 providers | 73 | 86.8%Hermes | 82.4% | 52.1%mean of 3 suites | $0.060$0.040 in · $0.120 out | 69 tok/s |
| 5 | Muse Spark OpenConsensus Protocol | 67 | 85.7%Codex | 76.6% | 92.1%mean of 3 suites | $1.05$0.600 in · $2.40 out | 15 tok/s |
| 6 | MiniMax M3.1 FlashImpCC | 66 | 92.0%ZCode | 73.2% | 71.1%mean of 3 suites | $0.262$0.150 in · $0.600 out | 45 tok/s |
| 7 | Ornith 1.5 35B A3BImpCC | 60 | 83.9%Hermes | 82.8% | 58.6%mean of 3 suites | $0.292$0.100 in · $0.870 out | 74 tok/s |
| 8 | Qwen3.8 27B2 providers | 59 | 77.4%Hermes | 85.1% | 63.9%mean of 3 suites | $0.688$0.150 in · $2.30 out | 56 tok/s |
| 9 | 59 | 87.5%Empryo | 79.9% | 65.1%mean of 3 suites | $0.237$0.150 in · $0.500 out | 69 tok/s | |
| 10 | GLM 5.3 Flash CybersecurityDealignAI · 2 providers | 59 | 72.4%OpenCode | 79.9% | 64.0%mean of 3 suites | $0.237$0.150 in · $0.500 out | 43 tok/s |
| 11 | DeepSeek V4 Flash 0731 Abliteratedcebeuq · Consensus Protocol | 57 | 77.8%Hermes | 77.3% | 33.8%mean of 3 suites | $0.175$0.140 in · $0.280 out | 43 tok/s |
| 12 | GLM 5.3 FlashConsensus Protocol | 53 | 70.0%ZCode | 81.5% | 53.7%mean of 3 suites | $0.175$0.100 in · $0.400 out | 41 tok/s |
Coding tasks solved by each model in each harness. Each cell reads, top to bottom: solved within 5 attempts, solved on the first try, then cost and median time per task.
| Model | Claude | Codex | DeepSeek Harness | Empryo | Hermes | Oh My Pi | OpenCode | Sleipnir | ZCode |
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | |||||||||
| DeepSeek V4.1 Flash 1017 UncensoredDealignAI | |||||||||
| MiMo V2.6 Flash RL | |||||||||
| MiniMax M3.1 Flash | |||||||||
| Nemotron 3.5 Super VL | |||||||||
| Ling 3.0 Flash | |||||||||
| Qwen3.8 Flash Next AbliteratedDealignAI | |||||||||
| Muse Spark Open | |||||||||
| gpt-oss-20b | |||||||||
| Gemma 4 31B | |||||||||
| Ornith 1.5 35B A3B | |||||||||
| DeepSeek V4 Flash 0731 Abliteratedcebeuq | |||||||||
| Qwen3.8 27B | |||||||||
| GLM 5.3 Flash CybersecurityDealignAI | |||||||||
| DiffusionGemma 26B A4B | |||||||||
| GLM 5.3 Flash | |||||||||
| Gork 27B | |||||||||
| Agents A1 4B Cybersecurityhotdogs |
Open a cell for that pair's cost, time, tokens and tool calls. A task that does not pass is attempted again, up to 5 attempts; figures are per task with every attempt included, and cost counts every call the harness made to the model.
All benchmarks · Equal-weight average · Higher scores are better
Recorded benchmark charges · USD / task · upper left is better
Full response time · seconds · upper left is better
DeepSeek: DeepSeek V4.1 Flash
9/9 benchmarks · Provisional · 99.9% mean dataset coverage · Includes: GPQA Diamond, Reasoning Gym, IFBench, AIME 2025, HMMT February 2026, BFCL v4, MRCR ≤32K, MRCR 32K–128K, MMLU-ProX Lite
85.6% · $0.00145Recorded / task · 115.3s
| Model / serving variant | Score | Cost / task | Time | Tested |
|---|---|---|---|---|
| 65% | $0.000949Recorded | 39.4s | 10/10 benchmarksComplete | |
| 59.6% | $0.0000597Recorded | 23.9s | 5/9 benchmarksProvisional | |
| 85.6% | $0.00145Recorded | 115.3s | 9/9 benchmarksProvisional | |
| 83.4% | — | 98.1s | 9/9 benchmarksComplete | |
| 81.7% | $0.000729Recorded | 267.8s | 8/8 benchmarksProvisional | |
| 72.5% | $0.00132Recorded | 100.1s | 9/9 benchmarksComplete | |
| 74% | $0.000304Recorded | 76.8s | 10/10 benchmarksComplete | |
| 52.6% | $0.0000648Recorded | 60.5s | 10/10 benchmarksComplete | |
| 89.9% | $0.00359Recorded | 256.5s | 9/9 benchmarksProvisional | |
| 76.2% | $0.00131Recorded | 83.7s | 10/10 benchmarksComplete | |
| 75% | $0.00168Recorded | 109.8s | 10/10 benchmarksComplete | |
| 52.2% | $0.000496Recorded | 51s | 6/9 benchmarksProvisional | |
| 72.9% | $0.00161Recorded | 99.3s | 10/10 benchmarksComplete | |
| 79% | $0.00433Recorded | 76.7s | 10/10 benchmarksComplete | |
| 78.4% | — | 236.5s | 9/9 benchmarksComplete | |
| 76.5% | $0.00121Recorded | 65.9s | 10/10 benchmarksComplete | |
| 71.4% | $0.00105Recorded | 141.9s | 10/10 benchmarksComplete | |
| 70.9% | $0.00102Recorded | 131.7s | 8/9 benchmarksProvisional | |
| 76.9% | $0.00107Recorded | 70s | 10/10 benchmarksComplete | |
| 76.1% | $0.00238Recorded | 109.7s | 9/9 benchmarksComplete |
Each benchmark has equal weight. Available provisional scores contribute; missing scores are omitted. Different benchmark mixes and tested samples can change the comparison. Cost and time average the same contributing benchmarks and remain unknown if any contribution is unmeasured.
Explore individual benchmarks
Benchmarks compare models as served, including their provider output limits. Individual views expose task evidence and request settings.
Benchmark scores and the evidence behind them. Each line represents one serving variant.
Measured accuracy · higher is better
Completed tasks / full dataset · farther means more evidence
Both scales run from 0 to 100%. Score spokes use the most represented protocol among the selected variants for each benchmark. Provider output limits are included as serving characteristics. Missing measurements leave gaps. Hollow score markers are provisional and can use different question samples. Dashed, unfilled outlines connect available provisional scores; gaps remain open. Dataset coverage measures tested tasks, not confidence or quality. Polygon area is not an overall rating.
Each column keeps its own metric. Brightness indicates a better score within that benchmark; provisional cells show the tested sample. Unscored cells distinguish collection progress, blocked runs and unavailable evaluators.
| Serving variant | GPQA DiamondAccuracy | Reasoning GymAccuracy | IFBenchPrompt-level loose accuracy | AIME 2025Accuracy | HMMT February 2026Accuracy | BFCL v4Overall accuracy | JSONSchemaBenchCompliance | MRCR ≤32KMatch ratio | MRCR 32K–128KMatch ratio | MMLU-ProX LiteAccuracy |
|---|---|---|---|---|---|---|---|---|---|---|
| Google: DiffusionGemma 26B A4BImpCC · NVFP4 | ||||||||||
| DeepSeek: DeepSeek V4.1 FlashConsensus Protocol · FP4 | ||||||||||
| Qwen: Qwen3.8 27BImpCC · NVFP4 | ||||||||||
| Ornith: Ornith 1.5 35B A3BImpCC · NVFP4 | ||||||||||
| Qwen: Qwen3.8 27BConsensus Protocol · FP8 | ||||||||||
| Consensus Protocol: Gork 27BImpCC · NVFP4 | ||||||||||
| InternScience: Agents A1 4B Cybersecurity (hotdogs)ImpCC · INT8 | ||||||||||
| Meta: Muse Spark OpenConsensus Protocol · NVFP4 | ||||||||||
| DeepSeek: DeepSeek V4.1 Flash 1017 Uncensored (DealignAI)Consensus Protocol · FP8 | ||||||||||
| NVIDIA: Nemotron 3.5 Super VLImpCC · NVFP4 | ||||||||||
| Qwen: Qwen3.8 Flash Next Abliterated (DealignAI)ImpCC · NVFP4 | ||||||||||
| Z.ai: GLM 5.3 Flash Cybersecurity (DealignAI)ImpCC · INT4 | ||||||||||
| Google: Gemma 4 31BConsensus Protocol · INT8 | ||||||||||
| MiniMax: MiniMax M3.1 FlashImpCC · NVFP4 | ||||||||||
| Xiaomi: MiMo V2.6 Flash RLImpCC · MXFP4 | ||||||||||
| Z.ai: GLM 5.3 Flash Cybersecurity (DealignAI)Consensus Protocol · INT4 | ||||||||||
| inclusionAI: Ling 3.0 FlashConsensus Protocol · FP8 | ||||||||||
| OpenAI: gpt-oss-20bConsensus Protocol · MXFP4 | ||||||||||
| inclusionAI: Ling 3.0 FlashImpCC · MXFP4 | ||||||||||
| DeepSeek: DeepSeek V4 Flash 0731 Abliterated (cebeuq)Consensus Protocol · FP8 | ||||||||||
| Z.ai: GLM 5.3 FlashConsensus Protocol · INT4 |
All benchmarks averages available scores equally within this inference type. Provisional scores contribute before a dataset is complete. Missing benchmarks are omitted and listed with each average; benchmark coverage is separate from the tested share of each dataset.
Only currently served configurations and measurements from the last 30 days qualify. Provisional scores appear after five scored questions and show the exact dataset coverage. These measurements describe a provider’s serving configuration, including its quantization.