Tag: benchmarks
A benchmark asks agents to size switchboards and check roof decks
Google and Mercor opened 29 of 85 engineering tasks to the public, and the best model still scores…
SemiAnalysis built a benchmark no frontier model can pass
The best agent on a new 206-task test across 11 domains scored 12 percent, and models broke a…
AgiBot proposes a robot metric built on hours rather than units
The Shanghai humanoid maker unveiled hardware, foundation models and a formula it hopes will replace shipment counts as…
DeepMind keeps Gemini 4 Carbon behind a cyber defenders gate
Google's strongest model ships first to a vetted cohort of cyberdefenders rather than the open developer market.
Jack Clark pulls his self-improving AI forecast forward a year
A benchmark measuring research taste shows capability doubling every three months, and the Anthropic co-founder noticed.
A study finds agent harnesses mostly buy you tokens
Two researchers held the model constant across three coding harnesses and found less than two points of difference…
IBM gets top customer service scores from notes, not new weights
ALTK-Evolve distills an agent's successes and failures into written rules, letting an older model beat newer entries on…
DeepSeek ships a Flash checkpoint and opens its own search API
The 1017 build posts 77.0 on DeepSWE, edging Claude Opus 5 in DeepSeek's own table, while search and…
Mistral’s 209M forecasting model tops charts without a GPU
Timestral is Mistral's first time-series family and its largest checkpoint takes first place across three public benchmarks from…
MiniMax confirms its stealth model was a Flash release all along
Space Bunny spent ten days at the top of OpenRouter's leaderboard before MiniMax revealed it as the new…
Six agents chasing a train refund turned aggressive fast
Given a hopeless refund claim, agents invented excuses, shared a client's medical history and sent 187 messages in…
New benchmark says agents pick badly on long tasks
Taste-Bench tests whether an agent chooses the better path when outcomes are hidden, and the strongest model managed…