AI Computer Vision & Graphics Machine Learning & Data Science Research

‘SketchTransfer’ Draws Understanding From Detail Invariance

A team of researchers from Mila and Google Brain believe simple pencil sketches could help AI models generalize to a better understanding of unseen images.

A team of researchers from Mila and Google Brain believe simple pencil sketches could help AI models generalize to a better understanding of unseen images.

Deep neural networks excel in practical perceptual tasks, and they keep getting stronger. As noted in Stanford’s AI Index 2019: “The time required to train a large image classification system on cloud infrastructure has fallen from about three hours in October 2017 to about 88 seconds in July 2019.”

Smart machines’ ability to generalize to variations not seen during model training however remains far from what humans are capable of. Humans quickly learn to pick up on salient qualities rather than looking at the little things. We can for example can identify and understand cartoon characters even in the absence of many visual details. Machine learning algorithms fail to generalize in such a way unless they have been explicitly trained to do so.


In a new paper, the Mila and Google Brain researchers introduce a “SketchTransfer” dataset and task wherein neural networks are evaluated on the quality of the abstractions they are able to learn without explicit supervision.


The SketchTransfer training dataset includes labeled real images from the CIFAR-10 dataset and unlabeled sketch images from the Quick Draw! dataset in categories such as frogs, birds, cats, dogs, cars, planes, etc.

SOTA models were tasked with recognizing and classifying objects in the sketches while provided only the labels for real images. The task is challenging because for example the only available clear difference between a dog sketch and a cat sketch might be the shape of the animal’s nose or ears. But instead of classifying images like humans would, a neural network might not immediately seek out this noticeable and distinctive feature.

Researchers observed that a SOTA technique which scored over 95 percent accuracy on MNIST to SVHN transfer was only able to manage 59 percent on the SketchTransfer task. While this was far better than random, it fell short of the 87 percent accuracy of a classifier trained directly on labeled sketches.


When humans look at the world we ignore most of the visual information and focus our attention on relevant details that represent invariant abstractions. Researchers conclude that teaching machines to see and understand the world in such a fundamental and humanlike way is approachable with contemporary methods, but that there is “substantial room for improvement.”

SketchTransfer offers the community a new tool for examining deep networks and their abilities to generalize abstractions.

The paper SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks is available on arXiv.

Journalist: Fangyu Cai | Editor: Michael Sarazen

6 comments on “‘SketchTransfer’ Draws Understanding From Detail Invariance

  1. Pingback: ‘SketchTransfer’ Draws Understanding From Detail Invariance #Detail_Invariance - NewsTakers

  2.  Quick Draw with Google online game allows you to put your creative idea into the neural network artificial intelligence.

  3. Yahoo mail login steps on the Yahoo mail login page,
    is as similar to other social media platform steps.
    To log in
    to your Yahoo account
    , Yahoo mail require some basic
    information from the users.

  4. Wapdam
    is an online web portal that offers a free download of
    Waptrick mp3 music, java games, videos, photos, mobile
    apps to your phone and computer.

  5. Cash App
    korean company LG has been a global household name for a long time; since well before cellphones were even a thing. Its five branches produce everything from home entertainment and appliances, vehicle components to mobile telephones and accessories.

  6. What a an amzing write up, after searching on the internet finally got what i’m looking for.
    Thanks very very much. Keep up the good work your doing here.

Leave a Reply to sportpaedia Cancel reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: