Image colourization is a practical task that automatically maps grey-scale images to plausible colour images. Such techniques enable the addition of natural colour to old family photos or historic images and can bring old films back to life for modern audiences. Although previous work in this area has shown impressive results, most approaches tend to struggle when the input images contain multiple objects, as this presents multiple plausible choices for object colourization and increases complexity and consistency challenges. To address this, researchers from National Tsing Hua University and Virginia Tech have introduced a novel deep learning framework for instance-aware colourization.
The research team proposes that colourization performance can be improved dramatically at the instance level for a few reasons. Learning to colourize instances is easier compared to existing methods that learn to colourize an entire image, which involves handling complex background clutter. Learning object-level representations from localized objects can also help avoid colour confusion with backgrounds.
The proposed method borrows the idea of instance-aware image synthesis and manipulation, providing a clear figure-ground separation and facilitating visual appearance synthesization and manipulation. The proposed instance-aware image colourization method can further advance the process by handling complex scenes with multiple instances and producing spatially coherent colourization results by tackling potentially overlapping instances simultaneously.
Given a grey-scale image X as input, the proposed method uses an off-the-shelf pre-trained model to detect object bounding boxes (Bi), so as to crop out instance Xi via Bi, then uses an instance colourization network trained end-to-end for instance and full-image colourization. To make the instance colours compatible with the predicted background colours, the researchers used a fusion module to fuse all the instance feature maps with the extracted full-image feature map to ensure globally consistent colourization results are obtained.
The results show that the proposed instance-aware model performs favourably when compared with existing state-of-the-art models, highlighting the potential for instance-aware colourization models to improve image colourization performance for complex images with multiple objects.
The paper Instance-aware Image Colorization is on arXiv.
Author: Hecate He | Editor: Michael Sarazen
0 comments on “Breakthrough Colourization Technique Enables Instance-Aware Treatment of Multiple Objects”