The ARC Prize Foundation seeks to develop metrics that test AI’s ability to adapt and perform tasks that mirror human intelligence, TechCrunch reported.
Greg Kamradt, a former Salesforce engineering director and founder of the AI product studio Leverage, will serve as the foundation’s president. Fundraising for the initiative is set to begin later this month.
Chollet’s new endeavor builds on his earlier work with ARC-AGI, the Abstract Reasoning Corpus for Artificial General Intelligence.
Introduced in 2019, ARC-AGI is a testing framework designed to evaluate whether AI systems can solve tasks that require reasoning beyond their training data. The benchmark presents puzzle-like challenges where AI must analyze grid patterns and generate solutions, emphasizing adaptability over rote memorization.
While ARC-AGI has highlighted the limitations of current AI systems, it has also underlined the significant gap between today’s technology and human-level reasoning. Even OpenAI’s highly advanced o3 model achieved qualifying scores on ARC-AGI only by leveraging vast computational resources.
Despite these achievements, Chollet has acknowledged ARC-AGI’s shortcomings, particularly its susceptibility to brute-force approaches by AI models. However, he remains optimistic that the benchmark can continue to push the boundaries of AGI research.
The ARC Prize Foundation plans to release an updated version of the ARC-AGI benchmark in early 2025, accompanied by a new competition to incentivize innovation in AGI.
Work on a third-generation benchmark has already begun, with the goal of refining how AGI progress is measured. Beyond technical benchmarks, the foundation aims to establish partnerships with leading AI labs and academic institutions to standardize AGI evaluation and foster collaboration within the field. OpenAI, which has used ARC-AGI to evaluate its systems, has hinted at potential future collaborations with Chollet’s team.
The concept of AGI itself remains contentious, and ARC-AGI has not escaped criticism. Some experts argue that it oversimplifies the idea of general intelligence by focusing too narrowly on abstract reasoning.
Moreover, the very definition of AGI is hotly debated. While some researchers suggest that AGI has already been achieved by systems outperforming humans in specific domains, Chollet views AGI as the moment when tasks “easy for humans but hard for AI” cease to exist. His vision aligns with the ARC Prize Foundation’s mission: to create benchmarks that measure progress toward this ambitious goal while inspiring innovation and collaboration across the industry.
