AI Adds Colour to Grandma’s Cherished Memories

After every Lunar New Year celebration, as the kids prepare to head back to the big city, grandmothers across China take their old tin of family photos out of the cupboard for a nostalgic trip down memory lane. Alas, these beloved old photographs are faded, torn, and monochrome.

Tencent Youtu — literally translated as “image optimization lab” — is the image processing, pattern recognition, machine learning, and data mining research arm of Chinese tech giant Tencent Group. Youtu is now offering a free retouching service for old photographs. Upload a scanned copy of your grandparents’ wedding picture on WeChat’s H5 built-in interactive pages, and Tencent’s AI will bathe the scene in natural colours.

The process begins with the AI detecting whether it is actually dealing with a monochrome photo. The team tells us many apparently black and white photographs are not truly monochromatic due to yellowing, molding, staining, and so forth. Colour contamination can also be also introduced during the scanning process if the scanner’s background is coloured.

The Youtu team analyzed millions of public stock photos and trained their algorithm to learn in multiple colorimetric spaces. The team separates the image based the HSV (Hue, Saturation, Value) colour model, which helps the algorithm identify moldy and damaged areas of sparse colour distributions precisely while ignoring greyish border areas. The team’s internal testing shows 95% image recognition accuracy.

The colouring process works as follows: the AI will tag different figures and objects in the photo. A trained neural network will then learn to “understand” tagged parts using semantic memories like “sky”, “grass”, “building”, and “face”, and finish the retouching with correct colour matches. AI-coloured images don’t restore the original colours, rather the algorithm is trained to come up with reasonable colour scenarios.

What if you have a old faded coloured photograph to begin with? Youtu’s AI can restore a colour photo by comparing it with similar images, then adjusting image intensities to enhance contrast using a technique called “histogram equalization.”

Richard Zhang’s team at University of California at Berkeley first offered the public an automatic photo colouring app in a project titled Interactive Deep Colorization, which is available for download on GitHub. Zhang’s 2016 paper Colourful Image Colorization further explains the network. For the curious, a history of automatic colouring AI can be traced back to Zezhou Cheng et al.’s 2015 work on Deep Colorization.

Tencent’s Youtu AI lab making the technology accessible to Chinese netizens is part of a developing “fun AI in 2018” trend, as Tencent’s H5 built-in apps make big bang developments. Last November, Tencent launched a built-in “AI experience center” that includes many fun and handy AI applications. The company also recently open-sourced WeChat’s built-in gaming and search functions to commercial developers.