AI Technology & Industry Review
On August 5, WeChat AI and Beijing Jiaotong University system developers released the paper WeChat Neural Machine Translation Systems for WMT21, revealing the architecture of their novel neural machine translation (NMT) system and the strategies they adopted to achieve impressive performance in the WMT21 competition.
A Google Research team draws inspiration from two numerical analysis methods — Hierarchical Matrix (H-Matrix) and Multigrid — to address the quadratic complexity problem of attention mechanisms in transformer architectures, proposing a hierarchical attention scheme that has linear complexity in run time and memory.
A research team from the University of Melbourne, Facebook AI, and Twitter Cortex proposes a black-box test method for assessing and debugging the numerical translation of neural machine translation systems in a systematic manner. The approach reveals novel types of errors that are general across multiple state-of-the-art translation systems.
A Google Research team proposes Wordcraft, a text editor with a built-in AI-powered creative writing assistant. Wordcraft uses few-shot learning and the natural affordances of conversation to support a variety of user interactions; and can help with story planning, writing and editing.
A research team from OpenAI proposes Codex, a specialized GPT model fine-tuned on publicly available code from GitHub that can produce functionally correct Python code bodies from natural language docstrings and could excel at a variety of coding tasks.
A research team from ByteDance AI Lab, University of Wisconsin–Madison and Nanjing University wins the ACL 2021 best paper award. Their proposed Vocabulary Learning via Optimal Transport (VOLT) approach leverages optimal transport to automatically find an optimal vocabulary without trial training.
A Google Research team proposes MergeDistill, a framework for merging pretrained teacher LMs from multiple monolingual/multilingual LMs into a single multilingual task-agnostic student LM to leverage the capabilities of the powerful language-specific LMs while still being multilingual and enabling positive language transfer.
A research team from Google proposes ByT5 architecture, a competitive token-free pretrained byte-to-byte transformer that can be straightforwardly adapted to process byte sequences without adding excessive computational cost.
A research team from UC Davis, Microsoft Research and Johns Hopkins University extends work on training massive amounts of linguistic data to reveal the grammatical structures in their representations to the domain of mathematical reasoning, showing that both the standard transformer and the TP-Transformer can compose the meanings of mathematical symbols based on their structured relationships.
A research team from Microsoft Research and Peking University peeps into pretrained transformers and investigates how factual knowledge is stored, proposing a method to identify “knowledge neurons,” which can be utilized to explicitly update and erase facts.
A research team from McGill University, Mila – Quebec AI Institute and Facebook AI proposes novel metrics and perturbation functions to detect, quantify and compare trade-offs between robustness and faithfulness in NMT systems, both on the corpus level and with particular examples.
An IBM research team proposes four multilingual adversarial attack strategies and attacks seven languages in a zero-shot setting on large multilingual pretrained language models (e.g. MBERT), reducing average performance by up to 85.6 percent.
A Yann LeCun team proposes dictionary learning to provide detailed visualizations of transformer representations and insights into semantic structures such as word-level disambiguation, sentence-level pattern formation, and long-range dependency captured by transformers.
The Beijing Academy of Artificial Intelligence (BAAI) releases Wu Dao 1.0, China’s first large-scale pretraining model.
A Microsoft research team provides concrete evidence showing that existing NLP models cannot robustly solve even the simplest of Math word problems, suggesting the hope that they might capably handle one-unknown arithmetic MWPs is untenable.
To track progress in natural language generation (NLG) models, 55 researchers from more than 40 prestigious institutions have proposed GEM (Generation, Evaluation, and Metrics), a “living benchmark” NLG evaluation environment.
Researchers from the University of Sheffield, Beihang University, and Open University’s Knowledge Media Institute have proposed a transfer learning approach that can automatically process historical texts at a semantic level to generate modern language summaries.
Researchers from Drexel University, Northeastern University and IT University Copenhagen explore how humans interact with AI in such contexts, with a focus on computer games.
Google Brain’s Switch Transformer language model packs a whopping 1.6 trillion parameters while effectively controlling computational cost. The model achieved a 4x pretraining speedup over a strongly tuned T5-XXL baseline.
A new model surpassed human baseline performance on the challenging natural language understanding benchmark.
OpenAI has trained a neural network called DALL·E that creates images from text captions for a wide range of concepts expressible in natural language.
Airbus has developed a new Smart Librarian (SL) FCOM QA system comprising a dialogue engine, retriever (search engine), and QA module.
In EMNLP 2020, a Montreal-based research team introduced a large medical text dataset designed to boost abbreviation disambiguation in the medical domain.
The Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) kicked off on Monday as a virtual conference.
A new study proposes using human feedback and interaction logs to boost offline reinforcement learning (RL) in natural language processing (NLP).
Amazon Alexa AI paper asks whether NLU problems could be mapped to question-answering (QA) problems using transfer learning.
Amazon extracts an optimal subset of architectural parameters for BERT architecture by applying recent breakthroughs in algorithms for neural architecture search.
Google recently introduced mT5, a multilingual variant of its “Text-to-Text Transfer Transformer” (T5), pretrained on a new Common Crawl-based dataset covering 101 languages.
Facebook AI open-sourced a multilingual machine translation (MMT) model that translates between any pair of 100 languages without relying on English data.
NeurIPS 2020 released its list of accepted papers this week with Google, Stanford, and MIT as the top affiliations.
Google AI has announced a new audiovisual speech enhancement feature in YouTube Stories (iOS) that enables creators to make better selfie videos by automatically enhancing their voices and reducing noise.
A team from Google, University of Cambridge, DeepMind, and Alan Turing Institute have proposed a new type of Transformer dubbed Performer, based on a Fast Attention Via positive Orthogonal Random features (FAVOR+) backbone mechanism.
The trimmed-down pQRNN extension to Google AI’s projection attention neural network PRADO compares to BERT on text classification tasks for on-device use.
Microsoft announced today that it has teamed up with OpenAI to exclusively license the AI research institute’s GPT-3 language model.
Facebook AI researchers and engineers just made live video content more accessible by enabling automatic closed captions for Facebook Live and Workplace Live.
OpenAI sets out to advance methods for training large-scale language models on objectives that more closely capture human preferences.
OpenAI researchers introduce GPT-f, an automated prover and proof assistant for the Metamath formalization language.
Researchers introduce a test covering topics such as elementary mathematics, designed to measure language models’ multitask accuracy.
Although OpenAI hasn’t yet officially announced the GPT-3 pricing scheme, Branwen’s sneak peek has piqued the interest of the NLP community.
AMBERT (A Multigrained BERT) leverages both fine-grained and coarse-grained tokenizations to achieve SOTA performance on English and Chinese language tasks.