LinkedIn Study Applies Deep NLP to Improve Search Systems
A LinkedIn research team evaluates deep natural language processing (NLP) on various representative search engine tasks to provide insights for the development of industry search engines.
Category: Research
Technical review of the newest machine intelligence research.
Apple Neural TTS System Study: Combining Speakers of Multiple Languages to Improve Synthetic Voice Quality
An Apple research team explores multiple architectures and training procedures to develop a novel multi-speaker and multi-lingual neural TTS system. The study combines speech from 30 speakers from 15 locales in 8 languages, and demonstrates that for the vast majority of voices, such multi-lingual and multi-speaker models can yield better quality than single speaker models.
Huawei Proposes Topic-Based Personalized Web Search Ranking, Integrating User Interests and Semantic Matching
A Huawei research team proposes a topic-based personalized ranking model (TPRM) that integrates pretrained contextualized term representations and user profiles constructed by a topic model to tailor the output ranking list.
UCSD & Microsoft Improve Image Recognition With Extremely Low FLOPs
A research team from University of California San Diego and Microsoft proposes Micro-Factorized Convolution (MF-Conv), a novel approach that can deal with extremely low computational costs (4M–21M FLOPs) and achieves significant performance gains over state of the art models in the low FLOP regime.
IBM Proposes Effective ML Drift Detection Via Weak Data Slices
An IBM research team leverages weak data slices to propose an effective method for drift detection in machine learning.
100+ Stanford Researchers Publish 200+ Page Paper on the AI Paradigm Shift Introduced by Large-Scale Models
In a 200+ page paper, Percy Liang, Fei-Fei Li, and over 100 other researchers from the Stanford University Center for Research on Foundation Models (CRFM) systematically describe the opportunities and risks of large-scale pretrained “foundation” models. The unique study aims to provide a clearer understanding of how these models work, when and how they fail, and the various capabilities provided by their emergent properties.
Logic Explained Deep Neural Networks: A General Approach to Explainable AI
A research team from Università di Firenze, Università di Siena, University of Cambridge and Universitè Côte d’Azur proposes a general approach to explainable artificial intelligence (XAI) in neural architectures, designing interpretable deep learning models called Logic Explained Networks (LENs). The novel approach yields better performance than established white-box models while providing more compact and meaningful explanations.
SFU & Tencent Explore the Production-Readiness of Learned Cardinality Estimation for DBMS
Simon Fraser University and Tencent researchers study cardinality estimation in database management systems and ask whether learned cardinality models are ready for deployment.
Tunable Stiffness Enables Fast and Efficient Swimming in Fish-like Robots
This paper uncovers the secrets of highly efficient swimming at varying speeds which could inform the design of next generation underwater drones.
Google Researchers Enable Transformers to Solve Compositional NLP Tasks
A Google Research team explores the design space of Transformer models in an effort to enable deep learning architectures to solve compositional tasks. The proposed approach provides models with inductive biases via design decisions that significantly impact compositional generalization, and achieves state-of-the-art results on the COGS and PCFG composition benchmarks.
Poison Ink: A Stealthy, Robust, General, Invisible and Flexible Backdoor Attack Method
A research team from the University of Science and Technology of China, Microsoft Cloud AI, City University of Hong Kong and Wormpex AI Research propose a robust and invisible backdoor attack called “Poison Ink” and demonstrates its immunity to state-of-the-art defence techniques.
WMT21 | Detailing WeChat AI & Beijing Jiaotong University’s NMT System Architecture
On August 5, WeChat AI and Beijing Jiaotong University system developers released the paper WeChat Neural Machine Translation Systems for WMT21, revealing the architecture of their novel neural machine translation (NMT) system and the strategies they adopted to achieve impressive performance in the WMT21 competition.
Tokyo U & Preferred Networks Propose a Fast Estimation Method for the Stability of Ensemble Feature Selectors
A research team from Tokyo University and Preferred Networks proposes a fast simulation-based method for estimating the stability of ensemble selectors.
Novel Feature Importance-Aware Transferable Adversarial Attacks Dramatically Improve Transferability
A research team from Zhejiang University, Wuhan University and Adobe Research proposes Feature Importance-Aware Attacks (FIA) that drastically improve the transferability of adversarial examples, achieving superior performance compared to state-of-the-art transferable attacks.
DeepMind’s Perceiver IO: A General Architecture for a Wide Variety of Inputs & Outputs
A DeepMind research team proposes Perceiver IO, a single network that can easily integrate and transform arbitrary information for arbitrary tasks while scaling linearly with both input and output sizes. The general architecture achieves outstanding results on tasks with highly structured output spaces, such as natural language and visual understanding.
The ‘Who’ in Explainable AI: New Study Explores the Creator-Consumer Gap
A team from the Georgia Institute of Technology, Cornell University and IBM Research conducts a mixed-methods study of how two different groups of “whos” — people with and without expert knowledge of AI — perceive different types of AI explanations.
Google’s H-Transformer-1D: Fast One-Dimensional Hierarchical Attention With Linear Complexity for Long Sequence Processing
A Google Research team draws inspiration from two numerical analysis methods — Hierarchical Matrix (H-Matrix) and Multigrid — to address the quadratic complexity problem of attention mechanisms in transformer architectures, proposing a hierarchical attention scheme that has linear complexity in run time and memory.
Google’s Generative Video Compression Technique Outperforms Traditional Neural Video Compression
A team from Google Research presents a neural video compression method based on generative adversarial networks (GANs) that outperforms previous neural video compression methods and advances generative video compression research.
DeepMind & Google Use Neural Networks to Solve Mixed Integer Programs
A team from DeepMind and Google Research leverages neural networks to automatically construct effective heuristics from a dataset for mixed integer programming (MIP) problems. The approach significantly outperforms classical MIP solver techniques.
Accelerating Quadratic Optimization Up to 3x With Reinforcement Learning
A research team from the University of California, Princeton University and ETH Zurich proposes RLQP, an accelerated QP solver based on operator-splitting QP (OSQP) that uses deep reinforcement learning (RL) to speed up the solver’s convergence rate.
Google & Northwestern U Present Provably Efficient Learning Algorithms for Neural Networks
A research team from Google Research and Northwestern University presents polynomial time and sample-efficient algorithms for learning an unknown depth-2 feedforward neural network with general ReLU activations, aiming to provide insights into whether efficient algorithms exist for learning ReLU networks.
MIT & Google Quantum Algorithm Trains Wide and Deep Neural Networks
A research team from MIT and Google Quantum AI presents a quantum algorithm for training classical neural networks in logarithmic time and provides numerical evidence of its efficiency on the standard MNIST image dataset.
Melbourne U, Facebook & Twitter Expose Novel Numerical Errors in NMT Systems
A research team from the University of Melbourne, Facebook AI, and Twitter Cortex proposes a black-box test method for assessing and debugging the numerical translation of neural machine translation systems in a systematic manner. The approach reveals novel types of errors that are general across multiple state-of-the-art translation systems.
DeepMind’s Epistemic Neural Networks Open New Avenues for Uncertainty Modelling in Large and Complex DL Systems
A research team from DeepMind presents epistemic neural networks (ENNs) as an interface for uncertainty modelling in deep learning, and proposes the KL divergence from a target distribution as a precise metric to evaluate ENNs.
Graph Kernel Attention Transformers: Toward Expressive and Scalable Graph Processing
A research team from Google Brain, Columbia University and University of Oxford proposes Graph Kernel Attention Transformers (GKATs), a new class of graph neural network that achieves greater expressive power than SOTA GNNs while reducing computation burdens.
Only Train Once: SOTA One-Shot DNN Training and Pruning Framework
A research team from Microsoft, Zhejiang University, Johns Hopkins University, Georgia Institute of Technology and University of Denver proposes Only-Train-Once (OTO), a one-shot DNN training and pruning framework that produces a slim architecture from a full heavy model without fine-tuning while maintaining high performance.
Google’s Wordcraft Text Editor Advances Human-AI Collaborative Story Writing
A Google Research team proposes Wordcraft, a text editor with a built-in AI-powered creative writing assistant. Wordcraft uses few-shot learning and the natural affordances of conversation to support a variety of user interactions; and can help with story planning, writing and editing.
DeepMind’s AlphaFold2 Predicts Protein Structures with Atomic-Level Accuracy
In a new paper published in the prestigious scientific journal Nature, DeepMind presents AlphaFold2, a redesigned neural-network system based on last year’s AlphaFold that can predict protein structures with atomic-level accuracy.
‘QuanTaichi’ Quantized Simulation: High Visual Quality With Reduced Memory Cost
A research team from Taichi Graphics, MIT CSAIL, Zhejiang University, Tsinghua University and Kuaishou Technology introduces a programming language and compiler for quantized simulation that achieves both high performance and significantly reduced memory costs by enabling flexible and aggressive quantization.
Google is Constructing Its Second “Cloud Region” in India
On July 15, tech conglomerate Google announced that it is broadening its cloud service infrastructure in India, by setting up a second “Cloud Region” in the Delhi-NCR area.
Baidu’s Knowledge-Enhanced ERNIE 3.0 Pretraining Framework Delivers SOTA NLP Results, Surpasses Human Performance on the SuperGLUE Benchmark
A research team from Baidu proposes ERNIE 3.0, a unified framework for pretraining large-scale, knowledge-enhanced models that can easily be tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning, and achieves state-of-the-art results on NLP tasks.
New Study Proposes Quantum Belief Function, Achieves Exponential Time Acceleration
A research team from the University of Electronic Science and Technology of China, Chinese Academy of Sciences, School of Education Shaanxi Normal University, Japan Advanced Institute of Science and Technology and ETH Zurich encodes the basic belief assignment (BBA) into quantum states and implements them on a quantum circuit, aiming to utilize quantum computation characteristics to better handle belief functions.
Human Evaluations No Longer the Gold Standard for NLG, Says Washington U & Allen AI Study
University of Washington and the Allen Institute for Artificial Intelligence researchers say human evaluations are no longer the gold standard for evaluating natural language generation models, as evaluators’ focus on surface-level text qualities degrades their ability to accurately assess current NLG models’ overall capabilities.
Two Lines of Code to Use a 2080Ti to Achieve What Was Previously Only Possible on a V100
As the dynamic computational graph is widely supported by many machine learning frameworks, GPU memory utilization for training on a dynamic computational graph becomes a key specification of these frameworks. In the recently released v1.4, MegEngine provides a way to reduce the GPU memory usage by additional computation using Dynamic Tensor Rematerialization (DTR) technique and further engineering optimization, which makes large batch size training on a single GPU possible.
OpenAI Fine-Tunes GPT-3 to Unlock Its Code Generation Potential for Difficult Problems
A research team from OpenAI proposes Codex, a specialized GPT model fine-tuned on publicly available code from GitHub that can produce functionally correct Python code bodies from natural language docstrings and could excel at a variety of coding tasks.
MIT Proposes Novel End-to-End Procedure for Corrupted Data Cleaning, Estimation, and Inference
A research team from MIT proposes a unified framework for estimation and inference in the presence of various forms of economic data corruption such as measurement errors, missing values, discretization, and differential privacy.
Back to School: MIT & UWaterloo Model Gets an ‘A’ on ML Course Problems
MIT and University of Waterloo researchers propose a machine learning model that outperforms the average student on problems from MIT’s 6.036 Introduction to Machine Learning course.
Meet CLIPDraw: Text-to-Drawing Synthesis via Language-Image Encoders Without Model Training
A research team from Cross Compass Ltd, Massachusetts Institute of Technology, Tokyo Institute of Technology and University of Tokyo presents CLIPDraw, an algorithm that synthesizes drawings based on natural language input without the need for any training.
ACL 2021 Best Paper: Finding the Optimal Vocabulary for Machine Translation via an Optimal Transport Approach
A research team from ByteDance AI Lab, University of Wisconsin–Madison and Nanjing University wins the ACL 2021 best paper award. Their proposed Vocabulary Learning via Optimal Transport (VOLT) approach leverages optimal transport to automatically find an optimal vocabulary without trial training.
Facebook & UC Berkeley Substitute a Convolutional Stem to Dramatically Boost Vision Transformers’ Optimization Stability
A research team from Facebook AI and UC Berkeley finds a solution for vision transformers’ optimization instability problem by simply using a standard, lightweight convolutional stem for ViT models. The approach dramatically increases optimizer stability and improves peak performance without sacrificing computation efficiency.