Automatic Image-to-Painting Translation Method Generates Vivid Paintings in Controllable Styles

Researchers from the University of Michigan, NetEase Fuxi AI Lab and Beihang University in China recently introduced “Stylized Neural Painter,” a novel automatic image-to-painting translation method that generates vivid and realistic artworks in controllable styles.

Painting is hard. In his lifetime, Dutch old master Johannes Vermeer — known for Girl with a Pearl Earring — completed a mere 34 canvases. Moreover, the many different painting mediums and styles create expressive possibilities ranging from Impressionist watercolours to Pop Art silkscreens and beyond. The rise of powerful generative modelling, image translation and style transfer techniques however is increasingly demonstrating the ability of AI and neural networks to mimic and even “create” such artistic images.

Existing image-to-image translation methods generally formulate the translation as a pixel-wise prediction or continuous optimization process in their pixel space. The new method instead treats this creative process in a vectorized environment, producing a sequence of physically meaningful stroke parameters that can be further used for rendering.

Since a typical vector render is not differentiable, the team designed a neural renderer which imitates the behaviour of the vector renderer, then frames the stroke prediction as a parameter searching process that maximizes the similarity between the input and the rendering output.

“Instead of manipulating each of the pixels in the output image, we simulate human painting behaviour and generate vectorized strokes sequentially with a clear physical significance,” the researchers explain. These generated stroke vectors can be further used for rendering with arbitrary output resolution.

Their method can “draw” in a variety of traditional and modern painting styles, including oil-paint brush, watercolour ink, marker pen and tape art. It can also be naturally embedded in a neural style transfer framework and jointly optimized to transfer its visual style based on different style reference images.

The researchers also identify a parameter coupling problem in previous neural renderers, and re-design their rendering process with a rasterization network and a shading network to better handle the disentanglement of shape and colour.

The team compared their approach with manually created methods and with Learning-to-Paint and SPIRAL, two new, stroke-based image-to-painting translation methods that train RL agents to paint. The results show the Stylized Neural Painter generates more vivid results and clearer distinction on brush textures. The paintings generated by the proposed method also have a high degree of fidelity in both global appearance and local textures.

The paper Stylized Neural Painting is on arXiv, and the code and animated results are available on the project GitHub.

---

Reporter: Yuan Yuan | Editor: Michael Sarazen

---

Synced Report | A Survey of China’s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic — 87 Case Studies from 700+ AI Vendors

This report offers a look at how China has leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon Kindle. Along with this report, we also introduced a database covering additional 1428 artificial intelligence solutions from 12 pandemic scenarios.

Click here to find more reports from us.

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.