AI Technology & Industry Review
HPC-AI Tech’s flagship open-source and large-scale AI system, Colossal-AI, now allows Hugging Face users to seamlessly develop their ML models in a distributed and easy manner.
A research team from DeepMind, Mila – University of Montreal and Google Brain proposes a neural network architecture that learns the graph structure of observational and/or interventional data via supervised training on synthetic graphs, making causal induction a black-box problem that generalizes well to new synthetic and naturalistic graphs.
In the new paper A Modern Self-Referential Weight Matrix That Learns to Modify Itself, a research team from The Swiss AI Lab, IDSIA, University of Lugano (USI) & SUPSI, and King Abdullah University of Science and Technology (KAUST) presents a scalable self-referential weight matrix (SRWM) that leverages outer products and the delta update rule to update and improve itself.
A Google Research team further explores the scaling approach for improving language modelling, leveraging the new Pathways distributed ML system to train a 540 billion parameter autoregressive transformer, Pathways Language Model (PaLM), that achieves state-of-the-art few-shot performance.
A research team from Sapienza University and OpenAI introduces an explanatory learning procedure that enables machines to understand existing explanations from symbolic sequences and create new explanations for unexplained phenomena, and further proposes Critical Rationalist Network (CRN) models for discovering explanations for novel phenomena.
A research team from University Medical Center Freiburg, ML Collective, and Google Brain introduces SimpleBits — an information-reduction method that learns to synthesize simplified inputs that contain less information yet remain informative for the task, providing a new approach for exploring the basis of network decisions.
A team from Google Research, University of Pennsylvania and Cornell University proposes a principled perspective to filter out common memorization for LMs, introducing “counterfactual memorization” to measure the expected change in a model’s prediction and distinguish “rare” (episodic) memorization from “common” (semantic) memorization in neural LMs.
A research team from Georgia Tech, Microsoft Research and Microsoft Azure AI studies the collections of “lottery tickets” in extremely over-parametrized models, revealing the generalization performance pattern of winning tickets and proving the existence of “super tickets.”
A research team from UC Davis, Microsoft Research and Johns Hopkins University extends work on training massive amounts of linguistic data to reveal the grammatical structures in their representations to the domain of mathematical reasoning, showing that both the standard transformer and the TP-Transformer can compose the meanings of mathematical symbols based on their structured relationships.
A research team from ETH Zürich presents an overview of priors for (deep) Gaussian processes, variational autoencoders and Bayesian neural networks. The researchers propose that well-chosen priors can achieve theoretical and empirical properties such as uncertainty estimation, model selection and optimal decision support; and provide guidance on how to choose them.
A research team from DeepMind explores how neural networks can be fused with algorithmic computation and demonstrates an elegant neural end-to-end pipeline that goes straight from raw inputs to general outputs while emulating an algorithm internally.
Twitter Chief Scientist Michael Bronstein, Joan Bruna from New York University, Taco Cohen from Qualcomm AI and Petar Veličković from DeepMind publish a paper that aims to geometrically unify the typical architectures of CNNs, GNNs, LSTMs, Transformers, etc. from the perspective of symmetry and invariance to build an “Erlangen Programme” for deep neural networks.
Researchers from Carnegie Mellon University, the University of Texas at Austin and Facebook AI propose a novel paradigm to optimize widths for each CNN layer. The method is compatible across various width optimization algorithms and networks and achieves up to a 320x reduction in width optimization overhead without compromising top-1 accuracy on ImageNet.
A research team from Microsoft Research and Peking University peeps into pretrained transformers and investigates how factual knowledge is stored, proposing a method to identify “knowledge neurons,” which can be utilized to explicitly update and erase facts.
A research team from Google and the University of California, Berkeley calculates the energy use and carbon footprint of large-scale models T5, Meena, GShard, Switch Transformer and GPT-3, and identifies methods and publication guidelines that could help reduce their CO2e footprint.
A team from Qualcomm AI proposes the direct mapping of a deep neural network onto an optical quantum computer through the language of quantum field theory, paving the way for the future development of novel quantum neural network architectures.
Researchers from Drexel University, Northeastern University and IT University Copenhagen explore how humans interact with AI in such contexts, with a focus on computer games.
Researchers from the University of Texas, Austin have proposed a novel framework called Training-Free Neural Architecture Search (TE-NAS) for “training-free” neural architecture search.
NVIDIA blog introduced company’s latest NeurIPS presentation: applying a novel neural network training technique, adaptive discriminator augmentation, to the popular NVIDIA StyleGAN2 model.
Researchers introduce a novel Physical Scene Graphs (PSG) approach designed to obtain a better structured understanding of visual scenes.
With AI models gaining power and momentum across a number of industries in recent years, meteorological researchers are now applying the tech in satellite data processing, nowcasting, typhoon and extreme weather forecasting and other business and environmental analytics areas.
Just as biologists gain insights into organisms by putting model specimens under their microscopes, AI Microscope was designed to help researchers analyze the features that form inside leading CV models.
Researchers proposed a “radioactive data” technique for subtly marking images in a dataset to help researchers later determine whether they were used to train a particular model.
To enable both content creators and end users to seriously restyle their apps’ interfaces while maintaining content detail clarity essential to their usability, researchers from Stanford have proposed ImagineNet, a novel and powerful new tool for interface customisation.
To help users design and tune machine learning models, neural network architectures or complex system parameters in an efficient and automatic way, in 2017 Microsoft Research began developing its Neural Network Intelligence (NNI) AutoML toolkit, open-sourcing v1.0 version in 2018.
Facebook AI Research team has introduced a new “point-based rendering” neural network module with an iterative subdivision algorithm that can integrate SOTA image segmentation models.
The ICLR 2020 conference programme chairs finally put the selection process behind them, announcing 687 out of 2594 papers had made it to ICLR 2020 — a 26.5 percent acceptance rate.
Synced Global AI Weekly December 1st
Rather than attempting to defend convolutional networks against them, the researchers introduce a novel enhanced adversarial training scheme, AdvProp, which treats adversarial examples as additional training examples to improve the accuracy of image classification models.
For better or worse, AI can now figure out what you’re doing even without “seeing” you. The MIT Computer Science & AI Lab (CSAIL) has unveiled a neural network model that can detect human actions through walls or in extremely dark places.
Facebook published a blog article demonstrating their newly developed code search tool, Neural Code Search (NCS).
Deep learning model performance has taken huge strides, allowing researchers to tackle tasks which were simply not possible for machines less than a decade ago.
Synced Global AI Weekly June 9th
Microsoft Research Asia (MSRA) has been dubbed the “Whampoa Academy for AI” in reference the elite Chinese military school. MSRA is a bootcamp for NLP research and has trained more than 500 interns, 20 PhDs and 20 postdocs over the past two decades.
San Francisco research company OpenAI has developed Sparse Transformer, a deep neural network which outperforms current state-of-the-art techniques for predicting long-sequence data in text, image and sound.
Synced Global AI Weekly April 28th
Neural networks for image recognition have matured from simple chain-like models to structures with multiple wiring paths. The emergence of Neural Architecture Search (NAS) can optimize models with more elaborate wiring and operation types.
DeepMind trained and tested its neural model by first collecting a dataset consisting of different types of mathematics problems. Rather than crowd-sourcing, they synthesized the dataset to generate a larger number of training examples, control the difficulty level and reduce training time.
Andrew Brock, first author of the high-profile research paper Large Scale GAN Training for High Fidelity Natural Image Synthesis (aka “BigGAN”), has posted a GitHub repository of an unofficial PyTorch BigGAN implementation that requires only 4-8 GPUs to train the model.
Facing the incomplete information environment, the asynchronous neural virtual self-play (ANFSP) method allows AI to learn to generate optimal decisions in multiple virtual environments. The approach has performed well in Texas Hold’em and multiplayer FPS video games.