Runta finds same-model coding costs vary 1.8 to 5.0 times by harness


With the same model, the harness running it made a substantial difference to average cost per task. Rankings moved a median of 11 of 22 places when models changed, and Runta estimated that 64.5% of spending on scored runs went to unsuccessful trials.
Runta and Baseten found that the highest average inference cost per task was 1.8 to 5.0 times the lowest when they kept the coding model fixed and changed the software running it. Inference is the process of running a model to generate an output.
That software, called a coding-agent harness, connects the model to its working environment and manages tool calls and the information supplied to the model. Runta’s earlier FrontierHarness Eval v1.0 compared harnesses using Kimi K3 alone; the new evaluation added models to test whether the choice of pairing mattered.
The New Stack said the largest same-model gap occurred with DeepSeek V4.1 Flash: Cline CLI cost five times as much per task as OpenCode v1, while its pass rate was two percentage points higher. Runta said differences of a few points were effectively a tie because each pass rate had a 95% interval of about ±10 points.
Across models, harnesses moved a median of 11 of 22 places in the ranking by average cost per task. Two examples show how sharply their positions changed:
| Harness | First model and rank | Second model and rank |
|---|---|---|
| Codex CLI | GLM-5.3: 1st | Kimi-K3: 16th |
| DeepSeek Harness: Creator | Muse Spark Open: 1st | GLM-5.3 Flash: 21st |

OpenAI says the new Sales plugin and new and expanded partner plugins can help sales teams research prospects, prepare pitches and update customer records.
The benchmark comprised 27,280 runs: 22 harness configurations worked on 62 terminal-based tasks with each of five open-weight models, making four attempts per combination. Baseten served GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse Spark Open and Kimi-K3. Open-weight models have model weights available for use outside their original developer.
Each harness used its default settings, the same version and settings across models, and maximum reasoning effort. Every trial ran in a fresh Runta environment that was deleted afterward, so it could not access files from earlier trials.
For average cost per task, Runta averaged the priced attempts for each task, then averaged those results across tasks. It calculated cost per successful task by dividing that average by the pass rate. A harness that is cheap per attempt can therefore cost more per verified success if it fails more often.
Runta estimated that 64.5% of inference spending on scored runs went to unsuccessful trials: 52.7% to failures and 11.8% to timeouts. Successful runs accounted for 35.5%. The estimates use Baseten’s published rates; Runta said another provider’s prices and cache could change the rankings.
Analysis
Runta attributed 92.8% of the variation in average cost per task, measured on a scale that treats each doubling equally, to differences between models. To test the same-model gaps, it shuffled trials between harnesses 300 times for each model; all observed cost ratios were larger than those produced by the shuffles (p = 0.003).
More steps per successful trial were associated with a higher cost per success, with a rank correlation of 0.51. The five best-value harnesses relied heavily on shell commands and used few subagents or summaries, but Runta did not test those design choices separately. The results do not establish that those choices caused the lower costs.
Runta recommends that teams test candidate harnesses with their intended model on a sufficiently large private set of their own tasks. It plans to add closed models and study harnesses that run continuously in the cloud on longer work.
1 min

The ruling lets Google proceed with buying internal records from the liquidated airline after it agreed to additional privacy protections.
2 min

California’s vehicle regulator restricted driverless freeway operations on 11 November, saying they posed an imminent public safety hazard. Waymo says it has paused freeway driving across its US markets while it evaluates performance.
2 min