That the range of genuinely viable options keeps growing. For a period, the landscape had a clear hierarchy, but today we see a set of models with different strengths. Claude, GPT, Gemini, and Grok occupy defensible positions with meaningfully different capabilities. DeepSeek's R1 release in January 2025 demonstrated that a focused team could produce reasoning capabilities approaching the best proprietary systems at a fraction of the training cost. Qwen has overtaken Llama in total downloads and become the most common base for fine-tuning. Models like Kimi, GLM, and Mistral's latest offerings have carved out niches where they outperform larger competitors on specific tasks.
The point is, working assumptions about which model performs best probably don't hold uniformly across different workflows. A model that excels at planning might stumble on implementation. A model that writes clean code might produce bloated, over-engineered architectures when given ambiguous requirements. A model optimized for deep reasoning might be slower than desired for routine tasks where iterative speed matters more than depth.
Research keeps confirming a gap between benchmark performance and real-world performance. Studies show that models achieving 90%+ on standardized tests like MMLU sometimes underperform in sustained production applications. This has prompted leaderboards like Vellum to exclude tests where top models have effectively maxed out the score. When every frontier model aces the same exam, that exam stops telling you which model will actually work for a particular codebase, particular prompts, or particular tasks.
The response to this proliferation of highly capable models and simultaneous benchmark collapse seems obvious: test the models yourself. Run actual tasks through multiple candidates. See which one works best for a specific context. The gap between benchmark performance and task-specific performance is an empirical question, and empirical questions have empirical answers.
In practice, this kind of experimentation has been difficult to sustain. The cognitive overhead adds up. You configure multiple providers, run parallel tests, wait while each model processes the task, manually compare outputs, iterate on the promising ones, and try to keep track of what you've learned. For developers with deadlines, this workflow competes with shipping; even developers who revisit model selection regularly tend to do so through informal testing and peer consensus rather than systematic comparison.
The tooling for rigorous model evaluation exists, but it's mostly designed for researchers and ML platform teams, not for practitioners trying to make grounded decisions while getting work done.
Two relatively recent shifts in the agentic coding landscape make empirical model experimentation more practical.
The first is technical: models have become capable enough to execute autonomously over extended tasks. They can follow a roadmap (especially in spec-driven development workflows) without constant supervision. When an agent has a specification—a structured plan that breaks work into discrete stages with clear acceptance criteria—it can run to completion without requiring active monitoring. You review finished work rather than babysitting output streams.
The second shift is in how we work with these long-running models. Spec-driven development, where each stage of a task is handled by a sequential agent with its own context window and external state tracking, creates natural boundaries for experimentation. The spec becomes a contract that any sufficiently capable model should be able to fulfill. The state tracking creates a paper trail that makes comparison meaningful. The staged execution means you can assess quality at each phase, not just at the end.
Together, these shifts mean you can run the same task across multiple model configurations and walk away, revisit, and compare the results. This is a fundamentally different experience than running a prompt through a playground and eyeballing the response.
Parallel model execution serves at least three distinct purposes, each valuable in its own right.
Some tasks don't have a single correct answer. Front-end design is the obvious example: given the same specification, different models will make different aesthetic choices, different structural decisions, different tradeoffs between simplicity and flexibility. Running a design task through multiple models generates options you wouldn't have thought to ask for.
This mode is about exploration. The model that produces the design you ship might not be the one that would win on a benchmark. Model diversity generates approaches that can be evaluated on their own merits.
Research on ensemble methods has demonstrated that when multiple models converge on the same answer, that convergence is meaningful. Studies on LLM ensembles in medical question-answering, for example, found that combining outputs from diverse models consistently outperformed the best individual model. The Iterative Consensus Ensemble approach showed accuracy improvements of 7-15 percentage points over single-model baselines. Ensemble methods reduce hallucination rates and improve factual accuracy because different models fail in different ways; their errors are less likely to align.
Our internal testing on code generation tasks reinforces this pattern and reveals something important about how you evaluate multiple solutions. When we scored each candidate solution independently, accuracy actually dropped below just picking the historically best-performing model. But when a selector agent reviewed all candidate solutions together, comparing them side-by-side, accuracy jumped to 75%, a 27.5 percentage point improvement over the single best model.
The difference came down to context. Isolated scoring evaluated each solution against an abstract rubric. Comparative evaluation revealed subtle semantic differences that looked identical when scored alone. In one task, the independent scorer gave the wrong solution 92/100 because it "looked correct." The comparative approach correctly identified that the solution was filtering data rather than restricting access—a distinction that only became apparent when both implementations were visible simultaneously.
The benefits scale with the number of candidates:
Filtered Dataset (60 tasks with mixed correct/incorrect)
You don't need to implement a formal ensemble system to benefit from this insight. If you run a task across five models and four of them converge on a similar architecture or implementation approach, that's an informal confidence signal. If they diverge wildly, that's a signal too, either about the models' relative weaknesses or about ambiguity in the specification that might need addressing before committing to any particular direction. Divergence often reveals that a task is underspecified in ways that weren't immediately apparent at the outset.
The cost differentials between models are now substantial. DeepSeek's API pricing runs roughly 20-50x cheaper than comparable OpenAI models for many tasks, a gap that OpenAI's own leadership has publicly acknowledged. Processing a million tokens through a frontier closed model might cost $10-15; the same volume through DeepSeek or a locally-run open-source model might cost under a dollar, or effectively nothing on your own hardware.
These economics create a genuine question: for which tasks is the expensive model actually worth it? The answer is almost certainly "some but not all." Routine code generation, boilerplate documentation, straightforward refactoring: these tasks might perform identically across models that differ by an order of magnitude in cost. Complex architectural decisions, subtle bug diagnosis, tasks requiring deep contextual understanding: these might genuinely benefit from frontier capabilities. But you won't know where particular workflows fall on this spectrum without testing.
There's a counterintuitive finding here too: smarter approaches aren't necessarily more expensive. In our testing, the comparative selection method that achieved 75% accuracy actually cost less per task ($3.97) than the inferior independent scoring approach ($5.73). The reflexive assumption that better results require more compute doesn't uniformly hold.
These approaches to experimentation rest in the hands os individual developers. But most software gets built by teams, and teams have a knowledge-sharing problem that individuals don't.
When model selection is purely informal, each developer on a team ends up with their own mental model of which tools work for what. One person swears by Claude for planning. Another has had better luck with GPT for debugging. A third tried DeepSeek once, had a bad experience, and wrote it off without realizing the issue was a poorly structured prompt rather than a model limitation. These mental models are often shaped by idiosyncratic experiences, and rarely compound into organizational knowledge.
Consider what an engineering team could learn if model selection became a shared, documented practice rather than a personal preference. Over the course of a quarter, a team of ten developers running parallel execution on varied tasks would generate substantial data about model performance in their specific context. Which models handle the codebase's idioms well? Which ones struggle with the domain's terminology? Where do the cost-performance tradeoffs actually land for these workloads?
The overhead for this kind of institutional knowledge doesn't need to be heavy. It could be as simple as a shared document that gets updated when someone discovers something non-obvious: "Qwen handles our GraphQL schema generation better than expected," or "Claude tends to over-abstract when working in the payments module." It could be a periodic sync where developers compare notes. It could be a lightweight internal tool that logs model choices and outcomes and surfaces patterns over time.
Deloitte's 2025 research on enterprise AI adoption found that 42% of organizations are still developing their strategy and 35% have no formal strategy at all. This is downstream of a simpler problem: most organizations aren't building systematic knowledge about how AI tools perform in their specific context. They're making decisions based on industry reputation, benchmark summaries, and the preferences of whoever set up the initial configuration. That's a fragile foundation for something that increasingly sits at the center of engineering workflows.
Empirical model selection at the individual level is useful. Empirical model selection that compounds into organizational knowledge provides additional strategic value.
The hierarchy that made model selection simple has given way to a field of different strengths where the "best" model depends heavily on the task, the context, and criteria that benchmarks don't capture.
For practitioners, this means model selection now both deserves and lends itself to more rigor than it typically gets; not the rigor of formal evaluation frameworks designed for researchers, but the practical experimentation of testing assumptions against actual work. Parallel execution makes this feasible in a way it wasn't before. Spec-driven architectures make the results meaningful rather than noisy.
Developers and teams who treat model selection as an ongoing empirical practice rather than a settled infrastructure choice will navigate the evolving landscape more effectively. They'll discover things about their workflows and their tools that single-model usage won't reveal. They'll build institutional knowledge that persists and compounds. And they'll make decisions grounded in evidence rather than reputation.