Google FixMatch: SOTA Performance on Semi-Supervised Learning Benchmarks

As the current go-to choice for computer vision applications, deep networks often achieve their strong performance through supervised learning — a method that requires labeled datasets. Despite AI’s many achievements and advancements over the years, the critical task of labeling data still falls to human experts. And they’re having a hard time meeting the demands of those data-hungry deep networks.

One solution to the expert shortage is to reduce model dependency on labeled data. Semi-supervised learning (SSL) aims to do that by coming up with ways to make use of unlabeled data during model training. And because unlabeled data can generally be obtained with minimal human labour, SSL’s performance boosts come at a relatively low cost.

A team from Google Research this week introduced FixMatch, an algorithm that combines two common SSL methods for deep networks: pseudo-labeling (aka self-training) and consistency regularization. Pseudo-labeling effectively uses the model’s class prediction as a label to train against. Consistency regularization meanwhile assumes that a model should output similar predictions when fed perturbed versions of the same image.

While FixMatch may seem but a simple combination of existing techniques, it nonetheless achieves SOTA performance across a variety of standard semi-supervised learning benchmarks, including a 94.93 percent accuracy on CIFAR-10 with 250 labels and an 88.61 percent accuracy with 40 (just four labels per class).

FixMatch first generates pseudo-labels using the model’s predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image.

“We also show how FixMatch can begin to bridge the gap between low-label semisupervised learning and few-shot learning—or even clustering: we obtain surprisingly-high accuracy with just one label per class,” the paper coauthors explain.

Due to the simplicity of FixMatch, the researchers were able to investigate nearly all aspects of the algorithm to explore how and why it works so well. They found that in order to get good results, especially in limited-label settings, certain design choices that had previously been underemphasized — such as weight decay or the choice of optimizers — can actually be very helpful in improving model performance.

The paper FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence is on arXiv. The code is available on the project GitHub.

---

Journalist: Yuan Yuan | Editor: Michael Sarazen