AI Machine Learning & Data Science Research

Google Brain’s SimCLRv2 Achieves New SOTA in Semi-Supervised Learning

Following on the February release of its contrastive learning framework SimCLR, the same team of Google Brain researchers guided by Turing Award honouree Dr. Geoffrey Hinton has presented SimCLRv2, an upgraded approach that boosts the SOTA results by 21.6 percent.

Following on the February release of its contrastive learning framework SimCLR, the same team of Google Brain researchers guided by Turing Award honouree Dr. Geoffrey Hinton has presented SimCLRv2, an upgraded approach that boosts the SOTA results by 21.6 percent.

Screenshot 2020-06-20 18.17.21.png
The proposed semi-supervised learning framework

The updated framework takes the “unsupervised pretrain, supervised fine-tune” paradigm popular in natural language processing and applies it to image recognition. Unlabelled data is learned in a task-agnostic way in the pretraining phase, which means the model has no prior classification knowledge. The researchers find that using a deep and wide neural network can be more label-efficient and greatly improve accuracy. Unlike SimCLR, whose largest model is ResNet-50, SimCLRv2’s largest model is a 152-layer ResNet, which is three times wider in channels and selective kernels.

Network size is the key at this and the following phase. Supervised labels are used in the fine-tuning stage to further refine accuracy. Here the team found that using fewer labelled examples helps the bigger and deeper network improve accuracy. The researchers also discovered that the task-specific prediction can be further distilled to a smaller network simply by labelling the unlabelled data again.

Screenshot 2020-06-20 18.18.16.png

The results on ImageNet are significant. SimCLRv2 achieves 79.8 percent top-1 accuracy, which is a 4.3 percent relative improvement over SimCLR, the previous SOTA. When fine-tuned on only 1 percent / 10 percent of labelled examples and distilled to the same architecture using unlabelled examples, the new framework achieves 76.6 percent / 80.9 percent top-1 accuracy, which is a 21.6 percent / 8.7 percent relative improvement. With distillation, these improvements can also be transferred to smaller ResNet-50 networks to achieve 73.9 percent / 77.5 percent top-1 accuracy using 1 percent / 10 percent of the labels.

The pretrained models for SimCLRv2 have not been released on GitHub yet, but the paper Big Self-Supervised Models are Strong Semi-Supervised Learners is on arXiv.


Author: Reina Qi Wan | Editor: Michael Sarazen

We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.

1 comment on “Google Brain’s SimCLRv2 Achieves New SOTA in Semi-Supervised Learning

  1. Pingback: Machine Learning: A Layman Overview – Academic Techie

Leave a Reply

Your email address will not be published. Required fields are marked *

%d