AI Technology & Industry Review
A Google AI research team explores zero-label learning (training with synthetic data only) in natural language processing, and introduces Unsupervised Data Generation (UDG), a training data creation procedure designed to synthesize high-quality training data without human annotations.
A new MIT CSAIL paper presents an alternative, hierarchical approach to sequence-to-sequence learning with quasi-synchronous grammars to improve sequence-to-sequence models’ compositional generalization on diagnostic tasks.
Researchers from Instituto de Telecomunicações, DeepMind, Institute of Systems and Robotics, Instituto Superior Técnico and Unbabel propose “∞-former” — a transformer model with unbounded long-term memory (LTM) that can attend to arbitrarily long contexts.
A Huawei research team proposes a topic-based personalized ranking model (TPRM) that integrates pretrained contextualized term representations and user profiles constructed by a topic model to tailor the output ranking list.
A Google Research team proposes Wordcraft, a text editor with a built-in AI-powered creative writing assistant. Wordcraft uses few-shot learning and the natural affordances of conversation to support a variety of user interactions; and can help with story planning, writing and editing.
A research team from Baidu proposes ERNIE 3.0, a unified framework for pretraining large-scale, knowledge-enhanced models that can easily be tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning, and achieves state-of-the-art results on NLP tasks.
A research team from OpenAI proposes Codex, a specialized GPT model fine-tuned on publicly available code from GitHub that can produce functionally correct Python code bodies from natural language docstrings and could excel at a variety of coding tasks.
A research team from Georgia Tech, Microsoft Research and Microsoft Azure AI studies the collections of “lottery tickets” in extremely over-parametrized models, revealing the generalization performance pattern of winning tickets and proving the existence of “super tickets.”
A research team from New York University, Facebook AI, and a CIFAR Fellow in Learning in Machines & Brains raise doubts regarding large-scale pretrained language models’ few-shot learning abilities. The researchers re-evaluate such abilities with held-out examples unavailable, which they propose constitutes “true few-shot learning.”
A research team from NVIDIA, Stanford University and Microsoft Research propose a novel pipeline parallelism approach that improves throughput by more than 10 percent with a comparable memory footprint, showing such strategies can achieve high aggregate throughput while training models with up to a trillion parameters.
A research team from Hugging Face shows that prompting is indeed beneficial for fine-tuning pretrained language models, and that this benefit can be quantified as worth hundreds of data points on average across classification tasks.
A research team from UC Berkeley, University of Maryland and UC Irvine identifies pitfalls that cause instability in the GPT-3 language model and proposes a contextual calibration procedure that improves accuracy by up to 30 percent.
A team from the Allen Institute for Artificial Intelligence and the University of Washington this week introduced TLDR generation, a new automatic summarization task for scientific papers.
Now, a team from Facebook AI Research, Inria, and Sorbonne Université have released CamemBERT, essentially a French version of Google AI’s game-changing pretrained language model BERT.
In a blog post today OpenAI today announced the final staged release of its 1.5 billion parameter language model GPT-2, along with all associated code and model weights.
Google’s new “ALBERT” language model has achieved state-of-the-art results on three popular benchmark tests for natural language understanding (NLU): GLUE, RACE, and SQuAD 2.0.
Synced Global AI Weekly February 24th
Github developer Hugging Face has updated its repository with a PyTorch reimplementation of the GPT-2 language model small version that OpenAI open-sourced last week, along with pretrained models and fine-tuning examples.