AI Technology & Industry Review
In the paper Detoxifying Language Models, a DeepMind research team critically discusses toxicity evaluation and mitigation for contemporary transformer-based English large language models and provides insights on safer model use and deployment.
A research team from Hugging Face introduces a block pruning approach targeting both small and fast models, which learns to eliminate full components of the original model while effectively dropping a large number of attention heads.
A research team from DeepMind proposes a bootstrapped meta-learning algorithm that overcomes the meta-optimization problem and myopic meta objectives, and enables the meta-learner to teach itself.
A new MIT CSAIL paper presents an alternative, hierarchical approach to sequence-to-sequence learning with quasi-synchronous grammars to improve sequence-to-sequence models’ compositional generalization on diagnostic tasks.
A research team from UC Berkeley takes a causal perspective on representation learning, formalizing non-spuriousness, efficiency and disentanglement representation learning desiderata using causal notions.
A research team from Carnegie Mellon University, Google Brain and UC Berkeley proposes a robust predictable control (RPC) method for learning reinforcement learning policies that use fewer bits of information. This simple and theoretically-justified algorithm achieves much tighter compression, is more robust, and generalizes better than prior methods, achieving up to 5× higher rewards than a standard information bottleneck.
A research team from Purdue University presents a self-adaptive algorithm for optimal deep neural network design. The adaptive network enhancement (ANE) method learns not only from given information (data, function, PDEs) but also from the current computer simulation.
MIT researchers present an automated, objective and transparent data-driven method for measuring media bias. The study analyses roughly a million articles from about a hundred newspapers for bias on various news topics, maps the newspapers into a two-dimensional media bias landscape, and shows that the data-driven results agree well with human-judgement classifications.
Researchers from Robotics at Google propose reformulating behavioural cloning using implicit models and show that this simple change can lead to remarkable improvements in performance across a wide range of contact-rich robot policy learning.
Researchers from Instituto de Telecomunicações, DeepMind, Institute of Systems and Robotics, Instituto Superior Técnico and Unbabel propose “∞-former” — a transformer model with unbounded long-term memory (LTM) that can attend to arbitrarily long contexts.
In the paper ReGen: Reinforcement Learning for Text and Knowledge Base Generation Using Pretrained Language Models, IBM researchers present ReGen, a bidirectional generation of text and graph that leverages reinforcement learning to push the performance of text-to-graph and graph-to-text generation tasks to a higher level.
A research team from The Swiss AI Lab IDSIA significantly improves the systematic generalization of transformer architectures, achieving accuracy up to 85 percent on the PCFG productivity split, and up to 81 percent on COGS.
A research team from Stanford University introduces BEHAVIOR, a benchmark for embodied AI with 100 realistic, diverse and complex everyday household activities in simulation. BEHAVIOR addresses challenges such as definition, instantiation in a simulator, and evaluation; and pushes the state-of-the-art by adding new types of state changes.
A research team from the University of Cambridge and Facebook AI proposes ProoFVer, a proof system for fact verification using natural logic. ProoFVer provides faithful explanations while outperforming state-of-the-art fact verification models.
A Nvidia research team presents Isaac Gym — a high-performance robotics simulation platform that runs an end-to-end GPU accelerated training pipeline. Compared to conventional RL training methods that use a CPU-based simulator and GPU for neural networks, Isaac Gym achieves training speedups of 2-3 orders of magnitude on continuous control tasks.
A DeepMind research team proposes Collect and Infer, a novel paradigm that explicitly models Reinforcement Learning (RL) as data collection and knowledge inference to dramatically boost RL data efficiency.
A team from Tsinghua University and Microsoft Research Asia proposes Fastformer, an efficient Transformer variant based on additive attention that achieves effective context modelling with linear complexity.
A Google Brain research team explores the internal representation structures of ViTs and CNNs on image classification tasks, providing insights on key differences between the two approaches.
A LinkedIn research team evaluates deep natural language processing (NLP) on various representative search engine tasks to provide insights for the development of industry search engines.
An Apple research team explores multiple architectures and training procedures to develop a novel multi-speaker and multi-lingual neural TTS system. The study combines speech from 30 speakers from 15 locales in 8 languages, and demonstrates that for the vast majority of voices, such multi-lingual and multi-speaker models can yield better quality than single speaker models.
A Huawei research team proposes a topic-based personalized ranking model (TPRM) that integrates pretrained contextualized term representations and user profiles constructed by a topic model to tailor the output ranking list.
A research team from University of California San Diego and Microsoft proposes Micro-Factorized Convolution (MF-Conv), a novel approach that can deal with extremely low computational costs (4M–21M FLOPs) and achieves significant performance gains over state of the art models in the low FLOP regime.
An IBM research team leverages weak data slices to propose an effective method for drift detection in machine learning.
In a 200+ page paper, Percy Liang, Fei-Fei Li, and over 100 other researchers from the Stanford University Center for Research on Foundation Models (CRFM) systematically describe the opportunities and risks of large-scale pretrained “foundation” models. The unique study aims to provide a clearer understanding of how these models work, when and how they fail, and the various capabilities provided by their emergent properties.
A research team from Università di Firenze, Università di Siena, University of Cambridge and Universitè Côte d’Azur proposes a general approach to explainable artificial intelligence (XAI) in neural architectures, designing interpretable deep learning models called Logic Explained Networks (LENs). The novel approach yields better performance than established white-box models while providing more compact and meaningful explanations.
Simon Fraser University and Tencent researchers study cardinality estimation in database management systems and ask whether learned cardinality models are ready for deployment.
A Google Research team explores the design space of Transformer models in an effort to enable deep learning architectures to solve compositional tasks. The proposed approach provides models with inductive biases via design decisions that significantly impact compositional generalization, and achieves state-of-the-art results on the COGS and PCFG composition benchmarks.
A research team from the University of Science and Technology of China, Microsoft Cloud AI, City University of Hong Kong and Wormpex AI Research propose a robust and invisible backdoor attack called “Poison Ink” and demonstrates its immunity to state-of-the-art defence techniques.
On August 5, WeChat AI and Beijing Jiaotong University system developers released the paper WeChat Neural Machine Translation Systems for WMT21, revealing the architecture of their novel neural machine translation (NMT) system and the strategies they adopted to achieve impressive performance in the WMT21 competition.
A research team from Tokyo University and Preferred Networks proposes a fast simulation-based method for estimating the stability of ensemble selectors.
A research team from Zhejiang University, Wuhan University and Adobe Research proposes Feature Importance-Aware Attacks (FIA) that drastically improve the transferability of adversarial examples, achieving superior performance compared to state-of-the-art transferable attacks.
A DeepMind research team proposes Perceiver IO, a single network that can easily integrate and transform arbitrary information for arbitrary tasks while scaling linearly with both input and output sizes. The general architecture achieves outstanding results on tasks with highly structured output spaces, such as natural language and visual understanding.
A team from the Georgia Institute of Technology, Cornell University and IBM Research conducts a mixed-methods study of how two different groups of “whos” — people with and without expert knowledge of AI — perceive different types of AI explanations.
A Google Research team draws inspiration from two numerical analysis methods — Hierarchical Matrix (H-Matrix) and Multigrid — to address the quadratic complexity problem of attention mechanisms in transformer architectures, proposing a hierarchical attention scheme that has linear complexity in run time and memory.
A team from Google Research presents a neural video compression method based on generative adversarial networks (GANs) that outperforms previous neural video compression methods and advances generative video compression research.
A team from DeepMind and Google Research leverages neural networks to automatically construct effective heuristics from a dataset for mixed integer programming (MIP) problems. The approach significantly outperforms classical MIP solver techniques.
A research team from the University of California, Princeton University and ETH Zurich proposes RLQP, an accelerated QP solver based on operator-splitting QP (OSQP) that uses deep reinforcement learning (RL) to speed up the solver’s convergence rate.
A research team from Google Research and Northwestern University presents polynomial time and sample-efficient algorithms for learning an unknown depth-2 feedforward neural network with general ReLU activations, aiming to provide insights into whether efficient algorithms exist for learning ReLU networks.
A research team from MIT and Google Quantum AI presents a quantum algorithm for training classical neural networks in logarithmic time and provides numerical evidence of its efficiency on the standard MNIST image dataset.
A research team from the University of Melbourne, Facebook AI, and Twitter Cortex proposes a black-box test method for assessing and debugging the numerical translation of neural machine translation systems in a systematic manner. The approach reveals novel types of errors that are general across multiple state-of-the-art translation systems.