Following on the February release of its contrastive learning framework SimCLR, the same team of Google Brain researchers guided by Turing Award honouree Dr. Geoffrey Hinton has presented SimCLRv2, an upgraded approach that boosts the SOTA results by 21.6 percent.
The updated framework takes the “unsupervised pretrain, supervised fine-tune” paradigm popular in natural language processing and applies it to image recognition. Unlabelled data is learned in a task-agnostic way in the pretraining phase, which means the model has no prior classification knowledge. The researchers find that using a deep and wide neural network can be more label-efficient and greatly improve accuracy. Unlike SimCLR, whose largest model is ResNet-50, SimCLRv2’s largest model is a 152-layer ResNet, which is three times wider in channels and selective kernels.
Network size is the key at this and the following phase. Supervised labels are used in the fine-tuning stage to further refine accuracy. Here the team found that using fewer labelled examples helps the bigger and deeper network improve accuracy. The researchers also discovered that the task-specific prediction can be further distilled to a smaller network simply by labelling the unlabelled data again.
The results on ImageNet are significant. SimCLRv2 achieves 79.8 percent top-1 accuracy, which is a 4.3 percent relative improvement over SimCLR, the previous SOTA. When fine-tuned on only 1 percent / 10 percent of labelled examples and distilled to the same architecture using unlabelled examples, the new framework achieves 76.6 percent / 80.9 percent top-1 accuracy, which is a 21.6 percent / 8.7 percent relative improvement. With distillation, these improvements can also be transferred to smaller ResNet-50 networks to achieve 73.9 percent / 77.5 percent top-1 accuracy using 1 percent / 10 percent of the labels.
The pretrained models for SimCLRv2 have not been released on GitHub yet, but the paper Big Self-Supervised Models are Strong Semi-Supervised Learners is on arXiv.
Author: Reina Qi Wan | Editor: Michael Sarazen
We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.
Pingback: Machine Learning: A Layman Overview – Academic Techie