Yoshua Bengio Team’s Large-Scale Analysis Reveals the Benefits of Modularity and Sparsity for DNNs

Deep neural networks (DNNs) have drawn much inspiration from the human cognitive process, evidenced recently in their incorporation of modular structures and attention mechanisms. By representing knowledge in a modular manner and selecting relevant information via attention mechanisms, DNN models can develop meaningful inductive biases, boost their out-of-distribution generalization abilities, and manipulate concepts at higher levels of cognition.

While modular architectures provide proven advantages for DNNs, there currently exists no rigorous quantitative assessment method for them due to the complexity and unknown nature of real-world data distributions. As such, it is unclear whether or to what extent the performance gains obtained by modular systems are actually attributable to good modular architecture design.

In the new paper Is a Modular Architecture Enough, a research team from Mila and the Université de Montréal conducts a rigorous and thorough quantitative assessment of common modular architectures that reveals the benefits of modularity and sparsity for DNNs and the sub-optimality of existing end-to-end learned modular systems.

The team summarizes their main contributions as: “

1. We develop benchmark tasks and metrics based on probabilistically selected rules to quantify two important phenomena in modular systems, the extent of collapse and specialization.
2. We distill commonly used modularity inductive biases and systematically evaluate them through a series of models aimed at extracting commonly used architectural attributes (Monolithic, Modular, Modular-op, and GT-Modular models).
3. We find that specialization in modular systems leads to significant boosts in performance when there are many underlying rules within a task, but not so much with only a few rules.
4. We find standard modular systems to be often sub-optimal in both their capacity on focusing on the right information as well as in their ability to specialize, suggesting the need for additional inductive biases. “

The team considers four model types with different levels of specialization: Monolithic, a large neural network that takes the entire data as input; Modular, a number of modules, each of which is a neural network that takes the data as input; Modular-op, similar to the modular system but with activation decided only by the rule context; and GT-Modular, which serves as an oracle benchmark, i.e., a modular system that specializes perfectly. They conduct a step-by-step analysis of the benefits of each system and contrast simple end-to-end trained modular systems with monolithic systems.

The team explores both in-distribution and out-of-distribution performance and evaluates how different models perform on a variety of tasks. They also introduce two metrics — Collapse-Avg and Collapse-Worst — to measure the amount of collapse suffered by a modular system; and use alignment, adaptation and inverse mutual information metrics to quantify the amount of specialization obtained.

In the experiments, the GT-Modular system generally had the highest performance, confirming the advantages of perfect specialization. Although standard end-to-end trained modular systems slightly outperformed monolithic systems, the team notes that these systems’ reliance on backpropagation of the task losses does not enable them to discover perfect specialization.

Both the Modular and Modular-op systems were shown to have collapse issues, but Modular-op generally suffered fewer. The team suggests a deeper investigation into forms of regularization may help alleviate these collapse problems.

Overall, this work shows that modular models outperform monolithic models. Although modular networks can obtain perfectly specialized solutions, end-to-end training does not recover them, and additional inductive biases are required to learn adequately specialized solutions. The team hopes their work will motivate future research into the design and development of modular architectures.

Open-sourced implementation is available on the project’s GitHub. The paper Is a Modular Architecture Enough? is on arXiv.

---

Author: Hecate He | Editor: Michael Sarazen

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.