AI Technology & Industry Review
Technical review of the newest machine intelligence research.
A research team from OneFlow and Microsoft takes a step toward automatic deep neural network structure design, exploring unsupervised structure-learning and leveraging the efficient coding principle, information theory and computational neuroscience to design structure learning without label information.
A research team from Google proposes ByT5 architecture, a competitive token-free pretrained byte-to-byte transformer that can be straightforwardly adapted to process byte sequences without adding excessive computational cost.
A research team from Google Cloud AI, Google Research and Rutgers University simplifies vision transformers’ complex design, proposing nested transformers (NesT) that simply stack basic transformer layers to process non-overlapping image blocks individually. The approach achieves superior ImageNet classification accuracy and improves model training efficiency.
A research team from Georgia Tech, Microsoft Research and Microsoft Azure AI studies the collections of “lottery tickets” in extremely over-parametrized models, revealing the generalization performance pattern of winning tickets and proving the existence of “super tickets.”
A research team from New York University, Facebook AI, and a CIFAR Fellow in Learning in Machines & Brains raise doubts regarding large-scale pretrained language models’ few-shot learning abilities. The researchers re-evaluate such abilities with held-out examples unavailable, which they propose constitutes “true few-shot learning.”
An IEEE team proposes AngularGrad — a novel optimization algorithm that takes both gradient direction and angular information into consideration. The method successfully reduces the zig-zag effect in the optimization trajectory and speeds up convergence.
A team from Cornell University and NTT Research proposes Physical Neural Networks (PNNs), a universal framework that leverages a backpropagation algorithm to train arbitrary, real physical systems to execute deep neural networks.
A research team from UC Davis, Microsoft Research and Johns Hopkins University extends work on training massive amounts of linguistic data to reveal the grammatical structures in their representations to the domain of mathematical reasoning, showing that both the standard transformer and the TP-Transformer can compose the meanings of mathematical symbols based on their structured relationships.
A research team from the University of Montreal and Max Planck Institute for Intelligent Systems constructs a reinforcement learning agent whose knowledge and reward function can be reused across tasks, along with an attention mechanism that dynamically selects unchangeable knowledge pieces to enable out-of-distribution adaptation and generalization.
A recent photo taken showing a Tesla Model Y with a LiDAR installed on the roof of the car has sparked heated discussion.
According to the “2021 Artificial Intelligence Development White Paper” released by Shenzhen AI Industry Association, in 2020, the scale of China’s key AI industry reached RMB 325.1 billion (USD 50.75 billion), a year-on-year increase of 16.7 percent.
On May 17, South Korea’s LG Group said it will invest USD 100 million over the next three years to develop advanced AI system with human-level learning, judgment, and thinking capabilities.
On May 24, Tsinghua University announced the establishment of quantum information class, led by Turing Award winner Professor Andrew Chi-Chih Yao as the chief professor.
A research team from ETH Zürich and Microsoft presents a systematic, comparative study of distributed ML training over serverless infrastructures (FaaS) and “serverful” infrastructures (IaaS), aiming to understand the system tradeoffs of distributed ML training with serverless infrastructures.
A research team from ETH Zürich presents an overview of priors for (deep) Gaussian processes, variational autoencoders and Bayesian neural networks. The researchers propose that well-chosen priors can achieve theoretical and empirical properties such as uncertainty estimation, model selection and optimal decision support; and provide guidance on how to choose them.
A research team from Adobe proposes Directional GAN (DGAN), a novel and simple approach for generating high-resolution images conditioned on expected semantic attributes, greatly simplifying the image content generating process for marketing campaigns, websites and banners.
A research team from Facebook shows how the power of transfer learning can enable pretraining on non-IDE, non-autocompletion and different-language example code sequences before fine-tuning on the autocompletion prediction task to improve model accuracy by over 50 percent on very small fine-tuning datasets and over 10 percent on 50k labelled examples.
A research team from Google proposes GSPMD, an automatic parallelism system for ML computation graphs that uses simple tensor sharding annotations to achieve different parallelism paradigms in a unified way, including data parallelism, within-layer model parallelism, spatial partitioning, weight-update sharding, optimizer-state sharding and pipeline parallelism.
A research team from Google shows that replacing transformers’ self-attention sublayers with Fourier Transform achieves 92 percent of BERT accuracy on the GLUE benchmark with training times seven times faster on GPUs and twice as fast on TPUs.
A research team from DeepMind explores how neural networks can be fused with algorithmic computation and demonstrates an elegant neural end-to-end pipeline that goes straight from raw inputs to general outputs while emulating an algorithm internally.
On May 11, Malaysian telecommunications company Axiata announced an investment of USD 60 million from SoftBank, towards its digital analytics and artificial intelligence arm ADA.
A research team from DeepMind and Onshape combines a general-purpose language modelling technique and an off-the-shelf data serialization protocol to propose a machine learning model that can automatically generate high-quality sketches for Computer-Aided Design.
A research team from ETH Zurich combines continual learning and self-supervision to propose a novel robot system that enables online life-long self-supervised learning of semantic scene understanding.
Imperial College London researchers show how to optimally train a variational quantum algorithm to represent quantum states and propose a stable variant of the quantum natural gradient, a generalized quantum natural gradient that can be trained free of barren plateaus.
A research team from MIT and MIT-IBM Watson AI Lab proposes Curious Representation Learning (CRL), a framework that learns to understand the surrounding environment by training a reinforcement learning (RL) agent to maximize the error of a representation learner to gain an incentive to explore the environment.
A research team from Facebook AI conducts a large-scale study on unsupervised spatiotemporal representation learning from videos. The work takes a unified perspective on four recent image-based frameworks (MoCo, SimCLR, BYOL, SwAV) and investigates a simple objective that can easily generalize unsupervised representation learning methodologies to space-time.
Twitter Chief Scientist Michael Bronstein, Joan Bruna from New York University, Taco Cohen from Qualcomm AI and Petar Veličković from DeepMind publish a paper that aims to geometrically unify the typical architectures of CNNs, GNNs, LSTMs, Transformers, etc. from the perspective of symmetry and invariance to build an “Erlangen Programme” for deep neural networks.
A research team from Huawei Noah’s Ark Lab and Tsinghua University proposes Extract Then Distill (ETD), a generic and flexible strategy for reusing teacher model parameters for efficient and effective task-agnostic distillation that can be applied to student models of any size.
Researchers from Carnegie Mellon University, the University of Texas at Austin and Facebook AI propose a novel paradigm to optimize widths for each CNN layer. The method is compatible across various width optimization algorithms and networks and achieves up to a 320x reduction in width optimization overhead without compromising top-1 accuracy on ImageNet.
A research team from NYU and Facebook proposes MDETR, an end-to-end modulated detector that identifies objects in images conditioned on a raw text query and is able to capture a long tail of visual concepts expressed in free-form text.
IBM and ETH Zurich researchers make progress in reconciling neurophysiological insights with machine intelligence, proposing a novel biologically inspired optimizer for artificial (ANNs) and spiking neural networks (SNNs) that incorporates synaptic integration principles from biology. GRAPES (Group Responsibility for Adjusting the Propagation of Error Signals) leads to improvements in the training time convergence, accuracy and scalability of ANNs and SNNs.
A research team from Google Research proposes small, fast, on-device disfluency detection models based on the BERT architecture. The smallest model size is only 1.3 MiB, representing a size reduction of two orders of magnitude and an inference latency reduction of a factor of eight compared to state-of-the-art BERT-based models.
A research team from Microsoft Research and Peking University peeps into pretrained transformers and investigates how factual knowledge is stored, proposing a method to identify “knowledge neurons,” which can be utilized to explicitly update and erase facts.
A research team from Google and the University of California, Berkeley calculates the energy use and carbon footprint of large-scale models T5, Meena, GShard, Switch Transformer and GPT-3, and identifies methods and publication guidelines that could help reduce their CO2e footprint.
A research team from McGill University, Mila – Quebec AI Institute and Facebook AI proposes novel metrics and perturbation functions to detect, quantify and compare trade-offs between robustness and faithfulness in NMT systems, both on the corpus level and with particular examples.
An IBM research team proposes four multilingual adversarial attack strategies and attacks seven languages in a zero-shot setting on large multilingual pretrained language models (e.g. MBERT), reducing average performance by up to 85.6 percent.
A research team from UC Berkeley and Carnegie Mellon University proposes a task-agnostic reinforcement learning method that reduces the task-specific engineering required for domain randomization of both visual and dynamics parameters.
A research team from Rice University, IBM and USC combine compressed sensing, non-convex optimization and acceleration techniques to introduce a new algorithm — Momentum Inspired Factored Gradient Descent (MiFGD) — that pushes QST beyond current capabilities.
A research team from DeepMind introduces Anakin and Sebulba, two architectures that demonstrate reinforcement learning platforms based on TPUs can efficiently deliver exceptional performance at scale and with low cost.
A research team from ETH Zurich leverages existing spike-based learning circuits to propose a biologically plausible architecture that is highly successful in classifying distinct and complex spatio-temporal spike patterns. The work contributes to the design of ultra-low-power mixed-signal neuromorphic processing systems capable of distinguishing spatio-temporal patterns in spiking activity.