Tag

Android Bench

All articles tagged with #android bench

technology20 days ago•88 min saved

Android Bench 2.0 pits AI against multi-day coding tasks as GPT-6 Astra leads

Google has launched Android Bench 2.0 to evaluate how well large language models and AI agents handle complex Android development tasks, including long-horizon projects that can take days to complete. The benchmark uses continuous scoring rather than binary pass/fail and includes tasks like upgrading dependencies, adding major features, and building apps from scratch. Early results show GPT-6 Astra leading with a 28% pass rate, while Gemini 3.8 Flash trails at 8%, illustrating the current landscape of AI coding capabilities.

via Android Central|

#ai #android #android-bench

technology3 months ago•4 min saved

Android Bench Expands with Eight New LLMs as Harbor Framework Rolls Out

Google’s Android Bench adds eight new LLMs (including Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, Qwen 3.7 Max) and switches to the Harbor testing framework, placing Gemini 3.1 Pro in fifth behind GPT-5.4 and Claude models; Fable 5 posts the top accuracy (~84.5%) but at a high cost per run, while Gemini 3.1 Pro is cheaper (~$87) and Gemini 3.5 Flash runs the most expensive and longest; Google invites developers to run benchmarks and submit results via GitHub to shape future Android Bench tasks.

via Ars Technica|

#android-bench #fable #gemini