LinkedIn Study Applies Deep NLP to Improve Search Systems
A LinkedIn research team evaluates deep natural language processing (NLP) on various representative search engine tasks to provide insights for the development of industry search engines.
Tag: NLP
Google Researchers Enable Transformers to Solve Compositional NLP Tasks
A Google Research team explores the design space of Transformer models in an effort to enable deep learning architectures to solve compositional tasks. The proposed approach provides models with inductive biases via design decisions that significantly impact compositional generalization, and achieves state-of-the-art results on the COGS and PCFG composition benchmarks.
Google’s H-Transformer-1D: Fast One-Dimensional Hierarchical Attention With Linear Complexity for Long Sequence Processing
A Google Research team draws inspiration from two numerical analysis methods — Hierarchical Matrix (H-Matrix) and Multigrid — to address the quadratic complexity problem of attention mechanisms in transformer architectures, proposing a hierarchical attention scheme that has linear complexity in run time and memory.
Melbourne U, Facebook & Twitter Expose Novel Numerical Errors in NMT Systems
A research team from the University of Melbourne, Facebook AI, and Twitter Cortex proposes a black-box test method for assessing and debugging the numerical translation of neural machine translation systems in a systematic manner. The approach reveals novel types of errors that are general across multiple state-of-the-art translation systems.
Google and UC Berkeley Propose Green Strategies for Large Neural Network Training
A research team from Google and the University of California, Berkeley calculates the energy use and carbon footprint of large-scale models T5, Meena, GShard, Switch Transformer and GPT-3, and identifies methods and publication guidelines that could help reduce their CO2e footprint.
Are Multilingual Language Models Fragile? IBM Adversarial Attack Strategies Cut MBERT QA Performance by 85%
An IBM research team proposes four multilingual adversarial attack strategies and attacks seven languages in a zero-shot setting on large multilingual pretrained language models (e.g. MBERT), reducing average performance by up to 85.6 percent.
Yann LeCun Team Uses Dictionary Learning To Peek Into Transformers’ Black Boxes
A Yann LeCun team proposes dictionary learning to provide detailed visualizations of transformer representations and insights into semantic structures such as word-level disambiguation, sentence-level pattern formation, and long-range dependency captured by transformers.
China’s GPT-3? BAAI Introduces Superscale Intelligence Model ‘Wu Dao 1.0’
The Beijing Academy of Artificial Intelligence (BAAI) releases Wu Dao 1.0, China’s first large-scale pretraining model.
Do NLP Models Cheat at Math Word Problems? Microsoft Research Says Even SOTA Models Rely on Shallow Heuristics
A Microsoft research team provides concrete evidence showing that existing NLP models cannot robustly solve even the simplest of Math word problems, suggesting the hope that they might capably handle one-unknown arithmetic MWPs is untenable.
Google Study Shows Transformer Modifications Fail To Transfer Across Implementations and Applications
A team from Google Research explores why most transformer modifications have not transferred across implementation and applications, and surprisingly discovers that most modifications do not meaningfully improve performance.
Transformers Scale to Long Sequences With Linear Complexity Via Nyström-Based Self-Attention Approximation
Researchers from the University of Wisconsin-Madison, UC Berkeley, Google Brain and American Family Insurance propose Nyströmformer, an adaption of the Nystrom method that approximates standard self-attention with O(n) complexity.
UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus
UmlsBERT is a deep Transformer network architecture that incorporates clinical domain knowledge from a clinical Metathesaurus in order to build ‘semantically enriched’ contextual representations that will benefit from both the contextual learning and domain knowledge.
Google Brain’s Switch Transformer Language Model Packs 1.6-Trillion Parameters
Google Brain’s Switch Transformer language model packs a whopping 1.6 trillion parameters while effectively controlling computational cost. The model achieved a 4x pretraining speedup over a strongly tuned T5-XXL baseline.
For Its Latest Trick, OpenAI’s GPT-3 Generates Images From Text Captions
OpenAI has trained a neural network called DALL·E that creates images from text captions for a wide range of concepts expressible in natural language.
NeurIPS 2020 | Teaching Transformers New Tricks
This year, 22 Transformer-related research papers were accepted by NeurIPS, the world’s most prestigious machine learning conference. Synced has selected ten of these works to showcase the latest Transformer trends.
Airbus AI Introduces Natural Language QA System for Flight Crews
Airbus has developed a new Smart Librarian (SL) FCOM QA system comprising a dialogue engine, retriever (search engine), and QA module.
EMNLP 2020 Best Paper Award Goes to UC Berkeley Team
The Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) kicked off on Monday as a virtual conference.
Google, Heidelberg University & NEC Propose Human Feedback for Real-World RL in NLP Systems
A new study proposes using human feedback and interaction logs to boost offline reinforcement learning (RL) in natural language processing (NLP).
Amazon Alexa AI’s ‘Language Model Is All You Need’ Explores NLU as QA
Amazon Alexa AI paper asks whether NLU problems could be mapped to question-answering (QA) problems using transfer learning.
Amazon’s BERT Optimal Subset: 7.9x Faster & 6.3x Smaller Than BERT
Amazon extracts an optimal subset of architectural parameters for BERT architecture by applying recent breakthroughs in algorithms for neural architecture search.
Google ‘mT5’ Pretrained Text-to-Text Transformer Achieves SOTA Performance on Multilingual Benchmarks
Google recently introduced mT5, a multilingual variant of its “Text-to-Text Transfer Transformer” (T5), pretrained on a new Common Crawl-based dataset covering 101 languages.
Facebook AI Model Directly Translates 100 Languages Without Using English Data
Facebook AI open-sourced a multilingual machine translation (MMT) model that translates between any pair of 100 languages without relying on English data.
Microsoft Gets Exclusive License for OpenAI’s GPT-3 Language Model
Microsoft announced today that it has teamed up with OpenAI to exclusively license the AI research institute’s GPT-3 language model.
New Multitask Benchmark Suggests Even the Best Language Models Don’t Have a Clue What They’re Doing
Researchers introduce a test covering topics such as elementary mathematics, designed to measure language models’ multitask accuracy.
Is OpenAI’s GPT-3 API Beta Pricing Too Rich for Researchers?
Although OpenAI hasn’t yet officially announced the GPT-3 pricing scheme, Branwen’s sneak peek has piqued the interest of the NLP community.
AMBERT: BERT with Multi-Grained Tokenization Achieves SOTA Results on English and Chinese NLU Tasks
AMBERT (A Multigrained BERT) leverages both fine-grained and coarse-grained tokenizations to achieve SOTA performance on English and Chinese language tasks.
Introducing Qian Yan, Baidu’s New Plan to Build 100 Chinese NLP Datasets in Three Years
Qian Yan, a plan to jointly develop the world’s largest Chinese natural language processing database.
Sepp Hochreiter on Parallels Between Attention Mechanisms and Modern Hopfield Networks
Researchers argue that “attention mechanism is the update rule of a modern Hopfield network with continuous states.”
Google Introduces NLP Model Understanding Tool
Google Research released a paper tackling this issue with a new open-source analytic platform: the Language Interpretability Tool (LIT).
How Smart is BERT? Evaluating the Language Model’s Commonsense Knowledge
Researchers dive deep into the large language model to discover how it encodes the structured commonsense knowledge it leverages on downstream commonsense tasks.
As Its GPT-3 Model Wows the World, OpenAI CEO Suggests ‘the Hype Is Way Too Much’
OpenAI’s 175 billion parameter language model GPT-3 has gone viral once again.
Grounded Language Learning: A Look at the Paper ‘Understanding Early Word Learning in Situated Artificial Agents’
This paper carefully engineers artificial-language learning experiments to replicate sources of information infants learn from and under what conditions.
Popular Microsoft Chatbot XiaoIce Gains Independence as a New Company Led by Di Li and Harry Shum
Microsoft announced it would spin off its chatbot business XiaoIce, with all associated technologies licensed to a newly formed independent company.
ACL 2020 Announces Its 779 Accepted Papers
Organizers of the 58th annual meeting of the Association for Computational Linguistics (ACL) on Sunday announced the list of accepted papers for the world-leading natural language processing (NLP) conference.
AI2 & UW Researchers Propose ‘TL;DR’ Paper Summarization Task
A team from the Allen Institute for Artificial Intelligence and the University of Washington this week introduced TLDR generation, a new automatic summarization task for scientific papers.
EMNLP 2020 Moved Entirely Online; Postponed One Week
EMNLP 2020 will be held entirely online from November 16 to 18 due to the worldwide spread of COVID-19.
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
Researchers propose a novel model compression approach to effectively compress BERT by progressive module replacing.
AI Tackles the 2019 Novel Coronavirus, Anemia and More; OpenAI Switches to PyTorch
Synced Global AI Weekly February 2nd
Facebook AI mBART: The Tower of Babel’s Silicon Solution
Facebook AI researchers have further developed the BART model with the introduction of mBART.
AraNet: New Deep Learning Toolkit for Arabic Social Media
A team of researchers from the Natural Language Processing Lab at the University of British Columbia in Canada have proposed AraNet, a deep learning toolkit designed for Arabic social media processing.