Neural Networks on the Brink of Universal Prediction with DeepMind’s Cutting-Edge Approach

Meta-learning stands out as a potent strategy to facilitate the rapid acquisition of new skills by AI systems, even with limited data. This methodology fosters the exploration of representations and learning approaches that can extend to unfamiliar tasks. Consequently, the construction of task distributions with ample breadth becomes imperative, ensuring that meta-learning models are exposed to a diverse array of structures and patterns. Such expansive exposure holds the promise of yielding “universal” representations, empowering systems to address a wide spectrum of problems and advancing us closer to achieving artificial general intelligence (AGI).

In a new paper Learning Universal Predictors, a Google DeepMind research team delves into the prospect of integrating Solomonoff Induction into neural networks through meta-learning. The paper proposes the utilization of Universal Turing Machines (UTMs) for generating training data, thereby enhancing meta-learning and enabling trained neural networks capable of mastering universal prediction strategies.

Solomonoff Induction (SI) serves as a compelling theoretical foundation for constructing an ideal universal prediction system, established by Solomonoff in 1964. A significant hurdle in implementing SI lies in identifying neural architectures and training data distributions that guide networks towards learning SI in the long run. Despite the theoretical capability of neural networks for universal computation, practical training methods, such as stochastic gradient descent, can constrain this potential.

To tackle this challenge, the research team opts for off-the-shelf architectures like Transformers and LSTMs while concentrating on devising an appropriate data training protocol. They leverage Universal Turing Machines (UTMs), which represent fully general computers, to generate data. Training on this “universal data” exposes the network to a wide array of computable patterns, steering it towards acquiring universal inductive strategies.

The team supplements their findings with a theoretical analysis of the UTM data generation process and training protocol, showcasing convergence to SI in the limit. Additionally, they conduct extensive experiments using various neural architectures (e.g., LSTMs, Transformers) and algorithmic data generators with differing complexities and degrees of universality.

The results indicate that large Transformers trained on UTM data successfully transfer their learning to other tasks, implying the acquisition of reusable universal patterns. On variable-order Markov sources, large LSTMs and Transformers demonstrate optimal performance, emphasizing their ability to model Bayesian mixtures over programs, a prerequisite for SI.

In summary, the study underscores the capability of neural models to implement algorithms and Bayesian mixtures, with larger models exhibiting heightened performance. Notably, networks trained on UTM data demonstrate transferability to other domains, suggesting the acquisition of a broad set of transferable patterns. The research team envisions enhancing future sequence models by scaling their approach using UTM data and integrating it with existing large datasets.

The code is on project’s GitHub. The paper Learning Universal Predictors on arXiv.

---

Author: Hecate He | Editor: Chain Zhang

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.