Together AI tests a cheaper-first route for coding SI

Its benchmark comparison found more successful fixes at lower cost when tests decided whether to call a second model. Reliable checks and longer waiting times shape the trade-off.

Its benchmark comparison found more successful fixes at lower cost when tests decided whether to call a second model. Reliable checks and longer waiting times shape the trade-off.
Trying a cheaper coding model before an expensive one produced more successful fixes at lower cost in Together AI’s undated benchmark comparison. The company reports that a DeepSeek V4 Pro 0813-first workflow solved 83.0% of DeepSWE tasks at $3.35 per task, compared with 72.7% at $8.37 for GPT-5.6 Sol alone.
The workflow uses tests to decide what happens next. DeepSeek attempts a fix first. If the tests reject its output, the task goes to Sol for another attempt. That limits use of the expensive model to the remaining tasks. It also gives those tasks a second chance, so the comparison is not simply between two ways of choosing a model for one attempt.
These are vendor-reported benchmark results, not demonstrated savings on a development team’s own code. Together explicitly conditions its recommendation on being able to verify the results.
DeepSWE is a benchmark of software-engineering tasks across different task types and programming languages. Together evaluated 113 tasks, with four trials per model per task, using each model’s maximum reasoning-effort setting. That produced 904 attempts in total.
On the benchmark’s single-attempt measure, called pass@1, Sol led with 72.7% success against DeepSeek’s 62.8%. Across four attempts, DeepSeek led at 88.5%, compared with Sol’s 85.8%. This measure, pass@4, asks whether at least one of four attempts passes the hidden test suite. It does not mean that every attempt succeeds.
That distinction separates reach from consistency. DeepSeek found a passing answer for more tasks when given several chances, while Sol solved 61 tasks on all four trials, compared with DeepSeek’s 35. The cheaper model’s wider coverage depended on retries and a way to identify the passing result.
Together reports a cost of $0.24 per DeepSeek attempt and $8.37 per Sol attempt in this run—a roughly 35-fold difference. But DeepSeek’s median attempt took 35 minutes, compared with 17 minutes for Sol. A cheaper-first route therefore trades a lower initial cost for a slower initial attempt, with another attempt needed when the first fails.
Tests also need to catch regressions: changes that break behavior that previously worked. Together says 20% of Sol’s failed attempts broke tests that had already passed, compared with 11% of DeepSeek’s failed attempts. Those percentages describe failures, not all attempts. Together recommends checking for regressions before accepting Sol’s changes; DeepSeek’s lower reported rate does not eliminate the need for that check.
Access needs care because the provider and endpoint—the model identifier used in an API request—can change which model runs. Together lists DeepSeek V4 Pro 0813 as available through its API under deepseek-ai/DeepSeek-V4-Pro-0813.
Separately, DeepSeek announced a September 14, 2026, 04:00 UTC start for redirecting requests to its own deepseek-v4-pro endpoint to V4.1-Flash, at V4.1-Flash rates, until V4.1-Pro launches. That announcement and Together’s listing concern different endpoints. Reproducing the comparison requires checking the actual model supplied, not relying on a similar name.
For a development team, the decisive question is whether its tests can reliably accept a correct fix, reject an incorrect one and detect broken existing behavior. Without comparable verification, a passing result on DeepSWE does not establish that the team’s own changes are safe. Where those checks exist, Together’s comparison offers a routing strategy to evaluate on real tasks—measuring successful fixes, total cost and waiting time together rather than choosing on model price alone.

The most serious flaw affects Windows, Linux and macOS and could let an attacker run code remotely. The company says it has found no evidence of active exploitation.
2 min

The invitation-only service lets artists offer album buyers up to four tracks before release. The wider release can still reach streaming services.
2 min

Meta says users must enable both Full Disk Access and its Messages connector. Journalist Jason Aten’s account conflicts with that explanation.
2 min