At NeurIPS 2024, while it was clear that state-of-the-art machine learning is still riding high on the success of large datasets and abundant computing power, signs are emerging that these approaches may be reaching their limits. Although computing power will likely continue to grow, the amount of publicly available data will not, and we may soon exhaust the most accessible corpora. For instance, on the AI engineering front, Ilya Sutskever and Sepp Hochreiter both foresee the end of purely data-driven approaches.
Sutskever envisions a future in which progress relies on agent-to-agent interactions and test-time inference—i.e., inference on a learned model at test time. Meanwhile, Hochreiter argues that as AI becomes more specialized and industrialized, integrating domain knowledge and structured priors will grow in importance, and that in some cases, model sizes might actually shrink.
This momentum away from brute-force data scaling was also evident in the keynote speakers’ focus on natural intelligence. This was emphasized by Francois Chollet creator of the Abstraction and Reasoning Corpus (ARC) challenge, scale is not sufficient for ARC he claimed, and systems that do inference through search and test time adaptation are key to solving the challenge.
"System-2 Reasoning at Scale"
Francoise Chollet, ARC-AGI prize
Alison Gopnik and Doina Precup both emphasized the power and efficiency of biological learning. Gopnik underscored how children learn dynamically and efficiently, while Precup highlighted continuous learning systems that adapt seamlessly over time.
"Understanding How Children Learn Can Help Us Understand And Improve AI"
Alison Gopnik, University of California, Berkeley
"RL has its limitations and we need continual learning like natural intelligence"
Doina Precup, DeepMind
"The future is Spatial Intelligence in 3D"
Fei-Fei Li, World Labs
Fei-Fei Li stressed the importance of adding spatial constraints to large-scale AI models, grounding them in the physical world’s structures and affordances. She argued that classification or generation alone is insufficient; models must also understand images in the context of a three-dimensional world and the possibilities for action it enables.
These themes were echoed by VERSES attendees. For example, the power of natural inference was spoken to by Karl’s keynote at the workshop on NeuroAI.
"The three face of AI" Keynote at the NeuroAI workshop
Karl Friston, VERSES
At the same workshop, Ran Wei gave a compelling talk on active exploration theory underpinned by the principle of expected free energy, and Tim Verbelen presented a poster on how to turn learning into an inference problem for robot control.
"Value of information and reward specification in active inference"
Ran Wei, VERSES
"Why learn when you can infer? Robot arm control with Hierarchical Active Inference"
Ran Wei and Tim Verbelen, VERSES
Conor Heins and Toon Van de Maele at the Bayesian decision making workshop showcased new sample-efficient Bayesian methods which enable continuous and uncertainty-aware learning.
Bayesian Decision Making Workshop
Conor Heins, VERSES
Francesco Innocenti and Tommaso Salvatori presented main-track posters on predictive approaches that advocate looking beyond backpropagation toward more biologically plausible, continuous process theories.
"Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?"
Francesco Innocenti, University of Sussex
"Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm"
Tomasso Salvatori, VERSES
Finally, joint work with Pietro Mazzaglia was presented in the main track on how to align and ground language embeddings with world models, allowing to express goals for an agent in natural language.
Overall, a pivot toward natural intelligence was clear—whether these will be recapitulated bottom up brute-force engineering approaches of mainstream ML, (i.e., Sutskever and Hochreiter) or, as VERSES has long argued, we need to step back and draw ideas from the natural systems around us, has yet to be determined.
We're betting on the latter.