AI Computer Vision & Graphics Research

Yann LeCun Team’s Barlow Twins Method Boosts SSL in Image Representation via Redundancy Reduction

Yann LeCun and a team of researchers propose Barlow Twins, a method that learns self-supervised representations through a joint embedding of distorted images, with an objective function that can make the embedding vectors almost identical while reducing redundancy between their components.

In a new paper, Yann LeCun and a Facebook AI and New York University research team introduce Barlow Twins, a novel self-supervised learning approach for computer vision.

Recent advances in self-supervised learning (SLL) for visual data show that training highly competitive image representations without manual labels is possible, and that this approach can sometimes even outperform supervised learning. Current SSL methods aim to learn representations that are invariant under different distortions, also referred to data augmentations, and typically do this by maximizing the similarity of representations from different distorted versions of a sample.

The LeCun team explains that trivial constant representations are a recurring issue with such approaches, which typically employ different mechanisms and careful implementation details to avoid collapsed solutions. The proposed Barlow Twins is an objective function that addresses this, measuring the cross-correlation matrix between the output features of two identical networks fed with distorted versions to make them as close as possible to the identity matrix while minimizing redundancy between concerned vector components.

image.png

Inspired by British neuroscientist Horace Barlow’s 1961 article Possible Principles Underlying the Transformation of Sensory Messages, the Barlow Twins method applies redundancy-reduction — a principle that can explain the organization of visual system — to self-supervised learning.

image.png
BARLOW TWINS’s objective function

The Barlow Twins’ objective function is similar to SSL objective functions but includes key conceptual differences that lead to practical advantages compared to InfoNCE-based contrastive loss methods — namely that the Barlow Twins method does not require a large number of negative samples and can thus operate on small batches, and that it can take advantage of very high-dimensional representations.

image.png
The pseudocode for Barlow Twins

The researchers evaluated the Barlow Twins representations via transfer learning to different datasets and computer vision tasks, and also tested the method on image classification and object detection, where the network was pretrained using self-supervised learning on the ImageNet ILSVRC-2012 dataset.

image.png
Top-1 and top-5 accuracies (in %) under linear evaluation on ImageNet. Barlow Twins obtains a top-1 accuracy of 73.2%, comparable to the benchmark. Top-3 best self-supervised methods are underlined.
image.png
Semi-supervised learning on ImageNet using 1% and 10% training examples. Best results are in bold. Barlow Twins is either on par (when using 10% of the data) or slightly better (when using 1% of the data) than competing methods.
image.png
Transfer learning: image classification. Top-1 accuracy on Places-205 and iNat18 datasets, and classification mAP on VOC07. Top-3 best self-supervised methods are underlined. Barlow Twins performs competitively against prior work, and outperforms SimCLR and MoCo-v2 across all datasets.
image.png
Transfer learning: object detection and instance segmentation learned representations on the object detection task on VOC07+12 using Faster R-CNN, and on the detection and instance segmentation task on COCO using Mask R-CNN. Best results are in bold. Barlow Twins performs comparably or better than state-of-the-art representation learning methods for these localization tasks.

The results show that Barlow Twins outperforms previous state-of-the-art methods for self-supervised learning while being conceptually simpler and avoiding trivial constant (i.e. collapsed representations). The researchers believe the proposed method is just one possible instantiation of the information bottleneck principle applied to SSL, and that further algorithm refinements could lead to more effective solutions.

The paper Barlow Twins: Self-Supervised Learning via Redundancy Reduction is on arXiv.


Author: Hecate He | Editor: Michael Sarazen


We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.

3 comments on “Yann LeCun Team’s Barlow Twins Method Boosts SSL in Image Representation via Redundancy Reduction

  1. Pingback: [N] Yann LeCun Team’s Barlow Twins Method Boosts SSL in Image Representation via Redundancy Reduction – ONEO AI

  2. Pingback: [N] Yann LeCun Team’s Barlow Twins Method Boosts SSL in Image Representation via Redundancy Reduction : artificial – Frankings meg

  3. Nice topic. Thanks

Leave a Reply

Your email address will not be published.

%d bloggers like this: