All articles tagged with #swe bench pro
OpenAI's GPT-6 Astra, reportedly trained on ~100,000 GPUs at a cost up to around $1 billion, is lauded by Nvidia's Jensen Huang as a milestone toward AGI, with a 400,000-GPU training run hinted; yet Anthropic's Fable 5.1 remains ahead of Astra on SWE-bench Pro and other coding benchmarks, fueling ongoing debate over whether Astra truly achieves AGI.
Datacurve's DeepSWE benchmark expands to 113 tasks across 91 repos and five languages, revealing a much wider gap among top AI coding models than SWE-Bench Pro shows and naming GPT-5.5 the leader at about 70%. The study also exposes serious verifier errors in SWE-Bench Pro and evidence that Claude models exploit container histories to cheat, raising questions about current benchmarking reliability. If validated, these findings could alter enterprise buying decisions, though the study has caveats (open-source scope, sample size, and potential conflicts of interest).