Releases

Scale Labs introduced repository-grounded rubrics for evaluating and reranking candidate software patches without executing tests.

Scale Labs introduced VeRO, a reproducible harness using versioned agent snapshots, controlled budgets, structured traces, and reference procedures.

Scale launched SWE Atlas, initially releasing Codebase QnA to evaluate how coding agents investigate and reason about real software systems.

Scale researchers released LHAW, a framework for generating underspecified long-horizon tasks and evaluating whether agents clarify ambiguity and recover performance.

Scale released MCP Atlas to evaluate how well AI agents combine tools across real Model Context Protocol servers.

Scale released a harder, contamination-resistant benchmark using real and commercial repositories to measure software agents on complex multi-file engineering tasks.

Scale published MASK as part of its safety research program for testing whether models knowingly state beliefs inconsistent with their internal representations.

Scale released FORTRESS to evaluate frontier-model safeguards against dual-use national-security and public-safety risks.

Scale and the Center for AI Safety published Humanity’s Last Exam, assembled from nearly 1,000 contributors across more than 500 institutions in 50 countries.

Scale and research collaborators introduced EnigmaEval, a multimodal reasoning benchmark built from novel puzzle-competition problems.

Scale released MultiChallenge to measure model performance on realistic multi-turn conversation problems involving memory, instruction retention, editing, and self-consistency.