On Tuesday, the French AI startup Mistral AI introduced Devstral 2, a coding model featuring 123 billion parameters, crafted to function as an integral part of an autonomous software engineering agent. The model secures a score of 72.2 percent on SWE-bench Verified, a benchmark that evaluates the capability of AI systems to address actual GitHub issues, positioning it among the leading open-weights models.
More significantly, Mistral didn’t limit its release to just an AI model; it also unveiled a new development application known as Mistral Vibe. This command line interface (CLI) resembles Claude Code, OpenAI Codex, and Gemini CLI, allowing developers to engage with the Devstral models directly from their terminal. This tool can analyze file structures and Git status to keep context consistent throughout an entire project, implement changes across various files, and execute shell commands autonomously. Mistral has released the CLI under the Apache 2.0 license.
It’s prudent to consider AI benchmarks with skepticism, yet we’ve gathered insights from personnel at major AI firms that they closely monitor model performance on SWE-bench Verified. This benchmark presents AI models with 500 genuine software engineering challenges sourced from GitHub issues in widely used Python repositories. The AI is required to comprehend the issue description, navigate the codebase, and produce a functional patch that passes unit tests. While some AI researchers have observed that about 90 percent of the tasks in the benchmark consist of relatively straightforward bug fixes that seasoned engineers can handle in under an hour, it remains one of the few standardized methods for comparing coding models.
Concurrent with the launch of the larger AI coding model, Mistral also debuted Devstral Small 2, a version with 24 billion parameters that achieved a score of 68 percent on the same benchmark and can operate locally on consumer-grade hardware such as a laptop without needing an Internet connection. Both models accommodate a 256,000 token context window, enabling them to process relatively large codebases (though whether one deems it large or small is quite subjective based on overall project complexity). The company released Devstral 2 under a modified MIT license and Devstral Small 2 under the more flexible Apache 2.0 license.
