NVIDIA Launch Web App for GauGAN2 to Generate Pictures Through Simple Phrases
On November 22, the NVIDIA blog introduced the interactive demo website app to generate photorealistic landscape images in real-time via text description.
Category: AI
Global machine intelligence updates.
Microsoft’s DeBERTaV3 Uses ELECTRA-Style Pretraining With Gradient-Disentangled Embedding Sharing to Boost DeBERTa Performance on NLU Tasks
Microsoft releases DeBERTaV3, improving the original DeBERTa model using ELECTRA-style pretraining with gradient-disentangled embedding sharing to achieve better pretraining efficiency and a significant performance jump.
Microsoft Asia’s Swin Transformer V2 Scales the Award-Winning ViT to 3 Billion Parameters and Achieves SOTA Performance on Vision Benchmarks
Microsoft Research Asia has upgraded their Swin Transformer with a new version featuring three billion parameters to train images with resolutions up to 1,536 x 1,536 and advance the SOTA on four representative vision benchmarks.
SPANN: A Highly-Efficient Billion-Scale Approximate Nearest Neighbour Search That’s 2× Faster Than the SOTA Method
A research team from Microsoft, Peking University, Tencent, and Baidu proposes SPANN, a simple but efficient memory-disk hybrid vector indexing and search system that guarantees both low latency and high recall and achieves a 2× speedup over the state-of-the-art nearest neighbour search (ANNS) solution while retaining the same recall quality and memory cost.
Intel’s Prune Once for All Compression Method Achieves SOTA Compression-to-Accuracy Results on BERT
An Intel research team presents Prune Once for All (Prune OFA), a training method that leverages weight pruning and model distillation to produce pretrained transformer-based language models with high sparsity ratios. Applied to BERT, the approach achieves state-of-the-art results in compression-to-accuracy ratio.
Is BERT the Future of Image Pretraining? ByteDance Team’s BERT-like Pretrained Vision Transformer iBOT Achieves New SOTAs
A research team from ByteDance, Johns Hopkins University, Shanghai Jiao Tong University and UC Santa Cruz seeks to apply the proven technique of masked language modelling to the training of better vision transformers, presenting iBOT (image BERT pretraining with Online Tokenizer), a self-supervised framework that performs masked prediction with an online tokenizer.
Google Brain & Radboud U ‘Dive Into Chaos’ to Show Gradients Are Not All You Need in Dynamical Systems
In the new paper Gradients Are Not All You Need, a Google Brain and Radboud University research team discusses a “particularly sinister” chaos-based failure mode that appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers.
Google’s Pet Portraits Will Find Art Doubles for Your Pets
Google recently has launched an adorable new feature for its Arts and Culture app named Pet Portraits that can compare your pet photo to artworks from museums worldwide and find their art doubles.
A Leap Forward in Computer Vision: Facebook AI Says Masked Autoencoders Are Scalable Vision Learners
In a new paper, a Facebook AI team advances autoencoding methods to the computer vision field and shows that masked autoencoders (MAE) are scalable self-supervised learners.
DeepMind’s One Pass ImageNet: A New Benchmark for Resource Efficiency in Deep Learning
A DeepMind research team presents the One Pass ImageNet (OPIN) problem, designed to study the space and compute efficiency of deep learning in a streaming setting with constrained data storage and to develop model training systems where each example is passed to the system only once.
Meta Teams up With Microsoft to Integrate Work App Workplace and Teams
On November 10, Meta (formerly known as Facebook) announced a partnership with another tech giant Microsoft to integrate its work app Workplace with Microsoft’s work-focused social network and collaboration software Teams.
Microsoft India Proposes Varuna: Scalable, Low-Cost Training of Massive Deep Learning Models
A Microsoft Research India team presents Varuna, a system for training massive deep learning models on commodity networking that eliminates the need for specialized hyperclusters and alleviates the cost, scale, and resource utilization challenges of deep learning model training.
Can ViT Layers Express Convolutions? Peking U, UCLA & Microsoft Researchers Say ‘Yes’
In the new paper Can Vision Transformers Perform Convolution?, a research team from Peking University, UCLA and Microsoft Research proves that a single ViT layer with image patches as the input can perform any convolution operation constructively, and show that ViT performance in low data regimes can be significantly improved using their proposed ViT training pipeline.
Introducing MetaICL: A Language Model Meta-Training Framework for Few-Shot In-Context Learning
A research team from the University of Washington, Facebook AI Research and the Allen Institute for AI introduces Meta-training for InContext Learning (MetaICL), a new meta-training framework for few-shot learning where an LM is meta-trained to learn in-context — conditioning on training examples to recover the task and make predictions.
Google & UC Berkeley’s Data-Driven Offline Optimization Approach Significantly Boosts Hardware Accelerator Performance, Reduces Simulation Time by More Than 90%
A research team from Google Research and UC Berkeley proposes PRIME, an offline data-driven approach that can architect hardware accelerators without any form of simulations. Compared to state-of-the-art simulation-driven methods, PRIME achieves impressive performance improvements of up to 1.54× while reducing the total required simulation time by up to 99 percent.
Washington U & Google Study Reveals How Attention Matrices Are Formed in Encoder-Decoder Architectures
In the new paper Understanding How Encoder-Decoder Architectures Attend, researchers from the University of Washington, Google Blueshift Team and Google Brain Team propose a method for decomposing hidden states over a sequence into temporal- and input-driven components, revealing how attention matrices are formed in encoder-decoder networks.
Clearview AI Ordered to Destroy All Facial Recognition Data Collected From Australia
On November 3, facial recognition company Clearview AI was ordered to destroy all existing images and templates the company collected from Australia and to cease collecting facial images and biometric templates from individuals in Australia.
Twitter Cortex Proposes LMSOC for Socially Sensitive Pretraining
In the new paper LMSOC: An Approach for Socially Sensitive Pretraining, a Twitter Cortex research team proposes a simple but effective approach for learning both linguistically contextualized and socially sensitive representations in large-scale language models.
Google Study Suggests Traditional Cost Indicator Reporting May Be a Misleading Measurement of Model Efficiency
In the new paper The Efficiency Misnomer, a Google Research team argues that no single cost indicator is sufficient for measuring model efficiency and offers suggestions to improve the reporting of efficiency metrics.
Facebook AI Is Teaching Robots to Perceive, Understand, and Interact Through Touch
On November 1, Facebook AI Research shared its progress on developing AI systems that can understand and interact through touch.
Warsaw U, OpenAI and Google’s Hourglass Hierarchical Transformer Model Outperforms Transformer Baselines
A team from the University of Warsaw, OpenAI and Google Research proposes Hourglass, a hierarchical transformer language model that operates on shortened sequences to alleviate transformers’ huge computation burdens.
DeepMind Study Resolves Delusions in Sequence Models for Interaction and Control
In the new paper Shaking the Foundations: Delusions in Sequence Models for Interaction and Control, a DeepMind research team explores the origins of mismatch problems in sequence models that lack understanding of the cause and effect of their actions, and addresses the problem by treating actions as causal interventions.
Softmax-free Vision Transformer With Linear Complexity: Achieving a Superior Accuracy/Complexity Trade-off
Researchers from Fudan University, University of Surrey and Huawei Noah’s Ark Lab identify the limitations of quadratic complexity for vision transformers (ViTs) as rooted in keeping the softmax self-attention during approximations. The team proposes the first softmax-free transformer (SOFT), which reduces the self-attention computation to linear complexity, achieving a superior trade-off between accuracy and complexity.
Google Open-Sources SCENIC: A JAX Library for Rapid Computer Vision Model Prototyping and Cutting-Edge Research
A research team from Google Brain and Google Research introduces SCENIC, an open-source JAX library for fast and extensible computer vision research and beyond. JAX currently supports implementations of state-of-the-art vision models such as ViT, DETR and MLP Mixer, and more open-sourced cutting-edge projects will be added in the near future.
Facebook AI’s NormFormer Employs Extra Normalization to Significantly Improve Transformer Pretraining
Facebook AI Research proposes NormFormer, an approach that improves pretraining perplexity and downstream task performance for both causal and masked language models, achieving GPT3-Large (1.3B) zero-shot performance 60 percent faster and improving fine-tuned GLUE performance by 1.9 percent.
Facebook AI Releases SaLinA: A Flexible and Simple Library for Learning Sequential Agents
A Facebook AI research team releases SaLinA, a reinforcement learning (RL) library for model-based RL, differentiable environments and multi-agent RL that simplifies the implementation of complex sequential learning models.
Deeper Is Not Necessarily Better: Princeton U & Intel’s 12-Layer Parallel Networks Achieve Performance Competitive With SOTA Deep Networks
In the new paper Non-deep Networks, a research team from Princeton University and Intel Labs argues it is possible to achieve high performance with “non-deep” neural networks, presenting ParNet (Parallel Networks), a novel 12-layer architecture that achieves performance competitive with its state-of-the-art deep counterparts.
DeepMind’s Fictitious Co-Play Trains RL Agents to Collaborate with Novel Humans Without Using Human Data
A DeepMind research team explores the problem of how to train agents to collaborate well with novel human partners without using human data and presents Fictitious Co-Play (FCP), a surprisingly simple approach designed to address this challenge.
Yann LeCun Team Challenges Current Beliefs on Interpolation and Extrapolation Regarding DL Model Generalization Performance
Facebook AI and NYU researchers challenge the conventional wisdom regarding interpolation in machine learning, arguing that interpolation almost never happens on high-dimensional datasets.
StyleNeRF: A 3D-Aware Generator for High-Resolution Image Synthesis with Explicit Style Control
In a paper currently under double-blind review for ICLR 2022, researchers propose StyleNeRF, a 3D-aware generative model that can synthesize high-resolution images at interactive rates while preserving high-quality 3D consistency, and can even generalize to unseen views with control on styles and poses.
Mention Memory: Incorporating Factual Knowledge From Various Sources Into Transformers Without Supervision
A research team from the University of Southern California and Google proposes TOME, a “mention memory” approach to factual knowledge extraction for NLU tasks. A transformer model with attention over a semi-parametric representation of the entire Wikipedia text corpus, TOME can extract information without supervision and achieves strong performance on multiple open-domain question answering benchmarks.
Google Proposes ARDMs: Efficient Autoregressive Models That Learn to Generate in any Order
A Google Research team introduces Autoregressive Diffusion Models (ARDMs), a model class encompassing and generalizing order-agnostic autoregressive models and discrete diffusion models that can generate variables in an arbitrary order and upscale variables.
Google Researchers Explore the Limits of Large-Scale Model Pretraining
A Google Research team conducts a systematic exploration comprising more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with parameters ranging from 10 million to 10 billion, evaluated on more than 20 downstream image recognition tasks, aiming to capture the nonlinear relationships between performance on upstream and downstream tasks.
ICCV 2021 Best Papers Announced
On October 13, ICCV 2021 announced its Best Paper Awards, honourable mentions, and Best Student Paper.
NVIDIA’s StyleGAN3 Is Fully Equivariant to Translation and Rotation, Improving GAN-Based Animation Generation
A NVIDIA and Aalto University research team presents StyleGAN3, a novel generative adversarial network (GAN) architecture where the exact sub-pixel position of each feature is exclusively inherited from the underlying coarse features, enabling a more natural transformation hierarchy and advancing GAN-based animation generation.
Are Patches All You Need? New Study Proposes Patches Are Behind Vision Transformers’ Strong Performance
A research team proposes ConvMixer, an extremely simple model designed to support the argument that the impressive performance of vision transformers (ViTs) is mainly attributable to their use of patches as the input representation. The study shows that ConvMixer can outperform ViTs, MLP-Mixers and classical vision models.
530 Billion Parameters! Microsoft and NVIDIA Trained the Largest Generative Language Model
On October 11, Microsoft introduced the largest and “the most powerful monolithic transformer language model” trained to date, a 530 billion parameter GPT-3-style generative language model.
China Has Won AI Battle With U.S., Says Ex-Pentagon Software Chief
According to Reuters, the Pentagon’s former software chief Nicolas Chaillan told the Financial Times on Sunday that “We have no competing fighting chance against China in 15 to 20 years. Right now, it’s already a done deal; it is already over, in my opinion.”
Apple Study Reveals the Learned Visual Representation Similarities and Dissimilarities Between Self-Supervised and Supervised Methods
An Apple research team performs a comparative analysis on a contrastive self-supervised learning (SSL) algorithm (SimCLR) and a supervised learning (SL) approach for simple image data in a common architecture, shedding light on the similarities and dissimilarities in their learned visual representation patterns.
Facebook & CMU’s Zero-Shot VideoCLIP Outperforms Fully-Supervised SOTA Methods for Video-Text Understanding
A research team from Facebook AI and Carnegie Mellon University presents VideoCLIP, a contrastive approach for pretraining a unified model for zero-shot video and text understanding without requiring annotated data for downstream tasks.