AI Technology

Fourier Feature Mapping Enables MLPs to Learn High-Frequency Functions in Low-Dimensional Domains

Researchers have proposed the use of Fourier feature mapping with MLPs in order to learn high-frequency functions in low-dimensional problem domains.

A multilayer perceptron (MLP) is a class of feed-forward neural networks with many perceptrons organized into layers. MLPs can solve problems stochastically and provide approximate solutions for complex problems such as fitness approximation.

A recent trend in computer vision is to replace the traditional discrete representations of objects, scene geometry, meshes and voxel grids with continuous functions parameterized by coordinate-based MLPs, which take low-dimensional coordinates as inputs and are trained to represent high-dimensional outputs (i.e. shape, density, colour, etc.). Coordinate-based MLPs are now widely used to represent a variety of visual signals, including images and 3D scenes.

Google Research and University of California researchers however found that standard MLPs are poorly suited for low-dimensional coordinate-based vision and graphics tasks and have difficulty learning high-frequency functions. To overcome these issues, they have proposed the use of Fourier feature mapping with MLPs in order to learn high-frequency functions in low-dimensional problem domains.

image.png

The researchers first leverage kernel regression with a neural tangent kernel (NTK) — which can describe the evolution of deep artificial neural networks during their training by gradient descent — to prove the deficiency of standard MLPs for low-dimensional coordinate-based vision and graphics tasks. Because the training points are distributed with uniform density, they then attempt to make the composed NTK shift-invariant. Finally, as a “wider” kernel with a slower spectral achieves faster training convergence for high-frequency component, their efforts to control the bandwidth of the NTK lead to the improvement of training speed and generalization.

image.png
Figure (a) visualizes the NTK function for a 4-layer ReLU MLP with one scalar input, where the kernel does not have a strong diagonal, making it perform poorly for kernel regression in low-dimensional problems. A basic input mapping makes the composed NTK stationary (b). A Fourier feature input mapping tunes the composed kernels’ width (c) and higher-frequency mappings, resulting in wider spectra of the composed kernels, enabling faster convergence for high-frequency components.
Effects of Fourier features on network convergence
Fourier features improve the results of coordinate-based MLPs for a variety of high-frequency low-dimensional regression tasks, i.e. image regression, 3D shape regression, MRI reconstruction and inverse rendering.

In experiments, the proposed Fourier feature mapping approach dramatically improves coordinate-based MLP performance across all tasks, with random Gaussian features performing best. The results illustrate the increasingly popular technique of using coordinate-based MLPs to represent 3D shapes in computer vision and graphics by using a simple mapping strategy.

The paper Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains is on arXiv.


Author: Hecate He | Editor: Michael Sarazen

We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.

Advertisements

0 comments on “Fourier Feature Mapping Enables MLPs to Learn High-Frequency Functions in Low-Dimensional Domains

Leave a Reply

Your email address will not be published.

%d bloggers like this: