AI Technology & Industry Review
In the new paper Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods, DeepMind and NYU Center for Neural Systems researchers introduce computational efficiency evaluation approaches designed to aid in the selection of optimal methods, datasets and models for pretraining visual tasks on a fixed FLOP budget.
In the new paper Toward a Realistic Model of Speech Processing in the Brain with Self-supervised Learning, researchers show that self-supervised architectures such as Wav2Vec 2.0 can learn brain-like representations from as little as 600 hours of unlabelled speech; and can also learn sound-generic and speech- and language-specific representations similar to those of the prefrontal and temporal cortices.
In the new paper Masked Autoencoders As Spatiotemporal Learners, a Meta AI research team extends masked autoencoders (MAE) to spatiotemporal representation learning for video. The novel approach introduces negligible inductive biases on space-time while achieving strong empirical results compared to vision transformers (ViTs) and outperforms supervised pretraining by large margins.
A DeepMind research team proposes ReLICv2, which demonstrates for the first time that representations learned without labels can consistently outperform a strong, supervised baseline on ImageNet and even achieve comparable results to state-of-the-art self-supervised vision transformers (ViTs).
In the NeurIPS 2021-accepted paper Self-Supervised Bug Detection and Repair, a Microsoft Research team proposes BUGLAB, a self-supervised approach that significantly improves on baseline methods for detecting bugs in real-life code.
In the new paper Masked Feature Prediction for Self-Supervised Visual Pre-Training, a Facebook AI Research and Johns Hopkins University team presents a novel Masked Feature Prediction (MaskFeat) approach for the self-supervised pretraining of video models that achieves SOTA results on video benchmarks.
In the new paper Understanding the World Through Action, UC Berkeley assistant professor in the department of electrical engineering and computer sciences Sergey Levine argues that a general, principled, and powerful framework for utilizing unlabelled data can be derived from reinforcement learning to enable machine learning systems leveraging large datasets to understand the real world.
An Apple research team performs a comparative analysis on a contrastive self-supervised learning (SSL) algorithm (SimCLR) and a supervised learning (SL) approach for simple image data in a common architecture, shedding light on the similarities and dissimilarities in their learned visual representation patterns.
An Oxford University research team presents PASS, a large (1.28M) image collection excluding humans, created as an ImageNet replacement for self-supervised pretraining without technical, ethical or legal issues.
A research team from ETH Zurich combines continual learning and self-supervision to propose a novel robot system that enables online life-long self-supervised learning of semantic scene understanding.
A research team from IBM introduces two systems for predicting information type: The TypeSuggest module, an unsupervised system designed to generate types for a set of seed query terms input by the user; and an Answer Type prediction module for predicting the correct answer type for user-provided questions.
Yann LeCun and a team of researchers propose Barlow Twins, a method that learns self-supervised representations through a joint embedding of distorted images, with an objective function that can make the embedding vectors almost identical while reducing redundancy between their components.
This year, NeurIPS is hosting two workshops dedicated to self-supervised learning: Self-Supervised Learning for Speech and Audio Processing on Friday, December 11; and Self-Supervised Learning — Theory and Practice on Saturday, December 12.
The SeLf-trAining framework for Distance mEtric learning (SLADE) framework combines self-supervised learning and distance metric learning methods to improve information retrieval performance.
New Spoken Language Understanding (SLU) research from MIT CSAIL and Amazon AI introduces step-skipping semi-supervised frameworks that take speech as input and achieve performance competitive to systems leveraging oracle text.
Facebook AI researchers have open-sourced the new wav2vec 2.0 algorithm for self-supervised language learning.
Enter Plan2Explore — a self-supervised RL agent designed to quickly generalize to unseen tasks in a zero or few-shot manner.
This summer Google Research extended BERT to video and introduced VideoBERT, a self-supervised learning model that can learn the relationship between videos and texts.
Frame-by-frame fine-grained labeling is however a time-consuming task. To tackle this problem, researchers from Google AI and DeepMind have introduced a novel self-supervised learning method called Temporal Cycle-Consistency Learning (TCC).
“AI is the best bot to keep people safe on our platforms,” Facebook Director of Artificial Intelligence Manohar Paluri told the F8 audience, adding that an effective way to achieve that goal is enabling Facebook’s AI system to “understand content and work effectively with less labeled training data.”