The deblurring of fast-moving objects is a longstanding challenge in photography and image processing. Although we have witnessed rapid progress in general image deblurring, most approaches require computational power beyond that of mobile phones and are unable to process high-resolution images with local motion.
In the new paper Face Deblurring Using Dual Camera Fusion on Mobile Phones, a Google team proposes Face Unblur, a novel system that uses a dual camera fusion technique to achieve reliable and robust face deblurring on mobile phones across diverse motion and lighting conditions.
The proposed system runs on Google Pixel 6 smartphones and addresses motion deblurring with the following goals:
- Reliably produce high-quality, high-resolution (2.3MP), realistic-looking, and artifact-free deblur results on moving subjects in the wild. We handle any form of face motion in the real world, where the motion blur size can reach hundreds of pixels.
- Require no additional user inputs other than pressing the shutter button. The system detects motion blur, identifies the blurry areas, and applies deblurring automatically.
- Run on mobile phone at interactive rate. In other words, users see the result immediately after the photo is taken.
- Operate within the power, memory, and computation budget provided by the SoC on Google Pixel 6.
- Support zero-shutter lag capture for motion subject. The deblurring is applied to the moment that the user intends to capture, not nearby frames.
The team’s technique focuses on the face in a captured image. It leverages the dual camera system available on modern premium phones and combines optical flow for alignment, a residual UNet for image fusion, and training on synthetic data to generate images with higher quality and resolution than state-of-the-art deblurring methods.
While some mobile phone cameras can sense movement in a subject and automatically increase shutter speed to reduce blurring, a faster shutter speed increases the noise in the captured image. The proposed approach exploits the dual camera setup on mobile phones, which typically includes both a wide-angle (W) main camera and a secondary ultrawide-angle (UW) camera. When users photograph a moving object with the W camera, the Face Unblur system also captures a synchronized reference image through the UW camera at a faster shuttle speed. The W image is thus low-noise but blurry, while the UW image is sharp but noisy.
With the W and UW raw bursts at hand, a trained deep convolutional neural network (CNN) aligns and fuses the W and UW captures to generate a clean and sharp linear image of the face region. The fused linear RAW is then fed to an image processing pipeline to output the final result.
The researchers also propose an adaptive stream system that dynamically enables or disables the UW sensor based on real-time motion analysis to reduce the computation burden. As such, the system can improve the generated image quality without increasing power or memory requirements.
The team evaluated Face Unblur performance on 1783 portrait images exhibiting motions under daily tasks such as jumping, walking and exercising. The results show the system outperforms state-of-the-art methods in single-image, multi-frame, face-specific, and video-based settings, and can generate more pleasing images than other systems.
The researchers believe theirs is the first mobile solution for effective face motion deblurring and hope their work will encourage further research on reference-based image deblurring.
The paper Face Deblurring Using Dual Camera Fusion on Mobile Phones is on arXiv.
Author: Hecate He | Editor: Michael Sarazen
We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.
0 comments on “Google Introduces the First Effective Face-Motion Deblurring System for Mobile Phones”