AI Technology & Industry Review
In the new paper DataMUX: Data Multiplexing for Neural Networks, a Princeton University research team proposes Data Multiplexing (DataMUX). The novel technique enables neural networks to process multiple inputs simultaneously and generate accurate predictions, increasing model throughput with minimal additional memory requirements.
A research team from Imperial College London and Microsoft Research introduces a cheap and accessible way to upgrade a popular off-the-shelf 3-axis 3D printer to 5 axes, enabling low-cost multi-axis and conformal 3D printing.
DeepMind researchers propose Hierarchical Perceiver (HiP), a model that retains the original Perceiver’s ability to process arbitrary modalities but is faster, can scale up to even more inputs/outputs, reduces the need for input engineering, and improves both efficiency and accuracy on classical computer vision benchmarks.
In the new paper Visual Attention Network, a research team from Tsinghua University and Nankai University introduces a novel large kernel attention (LKA) mechanism for an extremely simple and efficient Visual Attention Network (VAN) that significantly outperforms state-of-the-art vision transformers and convolutional neural networks on various computer vision tasks.
DeepMind trains agents to use keyboard and mouse commands with pixel and Document Object Model (DOM) observations to control computers, achieving state-of-the-art and human-level mean performance across all tasks on the MiniWob++ benchmark.
A research team from the University of Hong Kong, Shanghai AI Lab, Huawei Noah’s Ark Lab and the University of Washington takes dataset generation methods via large-scale pretrained language models (PLMs) to the extreme with ZEROGEN, a flexible and efficient zero-shot learning framework via dataset generation.
A research team from DeepMind and University College London proposes Neural Population Learning (NeuPL), an efficient and general framework that learns and represents diverse policies in symmetric zero-sum games within a single conditional network.
A team from Facebook AI Research, UC Berkeley and UCLA proposes Online Decision Transformers (ODT), an RL algorithm based on sequence modelling that incorporates offline pretraining and online finetuning in a unified framework and achieves performance competitive with the state-of-the-art models on the D4RL benchmark.
In the new paper Block-NeRF: Scalable Large Scene Neural View Synthesis, a team from UC Berkeley, Waymo and Google Research proposes Block-NeRF, a neural radiance fields variant capable of representing city-scale environments.
A Google Research team proposes Masked Generative Image Transformer (MaskGIT), a novel image synthesis paradigm that uses a bidirectional transformer decoder. MaskGIT significantly outperforms state-of-the-art transformer models on the ImageNet dataset and accelerates autoregressive decoding by up to 64x.
A Google Brain research team introduces EvoJAX, a JAX-based, scalable, general-purpose, hardware-accelerated neuroevolution toolkit that enables neuroevolution algorithms to work with neural networks running in parallel across multiple TPU/GPUs and achieves significant training speedups.
A team from Google Research and J.P. Morgan AI Research proposes an algorithm for scalable learning on multiplex networks with a large number of layers, improving efficiency over recent popular approaches.
A research team from Alibaba Group’s DAMO Academy proposes OFA (One For All), a pretrained model that unifies modalities and tasks to a simple Seq2Seq learning framework and achieves SOTA results on a series of multimodal tasks.
A research team from UC Berkeley, Amazon Web Services, Google, Shanghai Jiao Tong University and Duke University proposes Alpa, a compiler system for distributed deep learning on GPU clusters that automatically generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on the models they were designed for.
An OpenAI research team presents an expert iteration-based neural theorem prover capable of solving a curriculum of increasingly difficult mathematical problems (such as high-school olympiad-level problems) from a set of formal statements of sufficiently varied difficulty and without the need for associated ground-truth proofs.
A DeepMind research team presents AlphaCode, an automated code-generation system that can create novel solutions for programming problems that require deep reasoning and achieves a top 54.3% ranking in programming competitions.
A research team from Microsoft and NVIDIA leverages the NVIDIA Megatron-LM and Microsoft’s DeepSpeed to create an efficient and scalable 3D parallel system that combines data, pipeline, and tensor-slicing based parallelism, achieving superior zero-, one-, and few-shot learning accuracies and new state-of-the-art results on NLP benchmarks.
A research team from the University of Oxford proposes COIN++, a neural compression framework that addresses the existing issues of COIN while maintaining its generality and can seamlessly handle a wide range of data modalities.
A research team from Mila, Québec Artificial Intelligence Institute, Université de Montréal, CIFAR and IVADO Labs challenges the assumption that task diversity will improve model performance in meta-learning, finding instead that repeating the same tasks over the training phase can achieve performance similar to models trained on uniform sampling.