LinkedIn Study Applies Deep NLP to Improve Search Systems
A LinkedIn research team evaluates deep natural language processing (NLP) on various representative search engine tasks to provide insights for the development of industry search engines.
AI Technology & Industry Review
A LinkedIn research team evaluates deep natural language processing (NLP) on various representative search engine tasks to provide insights for the development of industry search engines.
A Google Research team explores the design space of Transformer models in an effort to enable deep learning architectures to solve compositional tasks. The proposed approach provides models with inductive biases via design decisions that significantly impact compositional generalization, and achieves state-of-the-art results on the COGS and PCFG composition benchmarks.
A Google Research team draws inspiration from two numerical analysis methods — Hierarchical Matrix (H-Matrix) and Multigrid — to address the quadratic complexity problem of attention mechanisms in transformer architectures, proposing a hierarchical attention scheme that has linear complexity in run time and memory.
A research team from the University of Melbourne, Facebook AI, and Twitter Cortex proposes a black-box test method for assessing and debugging the numerical translation of neural machine translation systems in a systematic manner. The approach reveals novel types of errors that are general across multiple state-of-the-art translation systems.
A research team from Google and the University of California, Berkeley calculates the energy use and carbon footprint of large-scale models T5, Meena, GShard, Switch Transformer and GPT-3, and identifies methods and publication guidelines that could help reduce their CO2e footprint.
An IBM research team proposes four multilingual adversarial attack strategies and attacks seven languages in a zero-shot setting on large multilingual pretrained language models (e.g. MBERT), reducing average performance by up to 85.6 percent.
A Yann LeCun team proposes dictionary learning to provide detailed visualizations of transformer representations and insights into semantic structures such as word-level disambiguation, sentence-level pattern formation, and long-range dependency captured by transformers.
The Beijing Academy of Artificial Intelligence (BAAI) releases Wu Dao 1.0, China’s first large-scale pretraining model.
A Microsoft research team provides concrete evidence showing that existing NLP models cannot robustly solve even the simplest of Math word problems, suggesting the hope that they might capably handle one-unknown arithmetic MWPs is untenable.
A team from Google Research explores why most transformer modifications have not transferred across implementation and applications, and surprisingly discovers that most modifications do not meaningfully improve performance.
Researchers from the University of Wisconsin-Madison, UC Berkeley, Google Brain and American Family Insurance propose Nyströmformer, an adaption of the Nystrom method that approximates standard self-attention with O(n) complexity.
UmlsBERT is a deep Transformer network architecture that incorporates clinical domain knowledge from a clinical Metathesaurus in order to build ‘semantically enriched’ contextual representations that will benefit from both the contextual learning and domain knowledge.
Google Brain’s Switch Transformer language model packs a whopping 1.6 trillion parameters while effectively controlling computational cost. The model achieved a 4x pretraining speedup over a strongly tuned T5-XXL baseline.
OpenAI has trained a neural network called DALL·E that creates images from text captions for a wide range of concepts expressible in natural language.
This year, 22 Transformer-related research papers were accepted by NeurIPS, the world’s most prestigious machine learning conference. Synced has selected ten of these works to showcase the latest Transformer trends.
Airbus has developed a new Smart Librarian (SL) FCOM QA system comprising a dialogue engine, retriever (search engine), and QA module.
The Conference on Empirical Methods in Natural Language Processing (EMNLP 2020) kicked off on Monday as a virtual conference.
A new study proposes using human feedback and interaction logs to boost offline reinforcement learning (RL) in natural language processing (NLP).
Amazon Alexa AI paper asks whether NLU problems could be mapped to question-answering (QA) problems using transfer learning.
Amazon extracts an optimal subset of architectural parameters for BERT architecture by applying recent breakthroughs in algorithms for neural architecture search.
Google recently introduced mT5, a multilingual variant of its “Text-to-Text Transfer Transformer” (T5), pretrained on a new Common Crawl-based dataset covering 101 languages.
Facebook AI open-sourced a multilingual machine translation (MMT) model that translates between any pair of 100 languages without relying on English data.
Microsoft announced today that it has teamed up with OpenAI to exclusively license the AI research institute’s GPT-3 language model.
Researchers introduce a test covering topics such as elementary mathematics, designed to measure language models’ multitask accuracy.
Although OpenAI hasn’t yet officially announced the GPT-3 pricing scheme, Branwen’s sneak peek has piqued the interest of the NLP community.
AMBERT (A Multigrained BERT) leverages both fine-grained and coarse-grained tokenizations to achieve SOTA performance on English and Chinese language tasks.
Qian Yan, a plan to jointly develop the world’s largest Chinese natural language processing database.
Researchers argue that “attention mechanism is the update rule of a modern Hopfield network with continuous states.”
Google Research released a paper tackling this issue with a new open-source analytic platform: the Language Interpretability Tool (LIT).
Researchers dive deep into the large language model to discover how it encodes the structured commonsense knowledge it leverages on downstream commonsense tasks.
OpenAI’s 175 billion parameter language model GPT-3 has gone viral once again.
This paper carefully engineers artificial-language learning experiments to replicate sources of information infants learn from and under what conditions.
Microsoft announced it would spin off its chatbot business XiaoIce, with all associated technologies licensed to a newly formed independent company.
Organizers of the 58th annual meeting of the Association for Computational Linguistics (ACL) on Sunday announced the list of accepted papers for the world-leading natural language processing (NLP) conference.
A team from the Allen Institute for Artificial Intelligence and the University of Washington this week introduced TLDR generation, a new automatic summarization task for scientific papers.
EMNLP 2020 will be held entirely online from November 16 to 18 due to the worldwide spread of COVID-19.
Researchers propose a novel model compression approach to effectively compress BERT by progressive module replacing.
Synced Global AI Weekly February 2nd
Facebook AI researchers have further developed the BART model with the introduction of mBART.
A team of researchers from the Natural Language Processing Lab at the University of British Columbia in Canada have proposed AraNet, a deep learning toolkit designed for Arabic social media processing.