Japanese Manga Translation Via Multimodal Context-Aware Framework

A new machine translation method from Japanese start-up Mantra Inc., Yahoo Japan and the University of Tokyo enables global manga fans to enjoy immediate translations of their favourite Japanese comics. The researchers say this is the first comprehensive system for fully automated manga translation from the original Japanese into English or Chinese.

Comics are an integral component of many cultures. US superhero sagas, Korean webtoons and Chinese manhua all have unique styles and characteristics. Most comics however remain readily available only in their domestic markets, due largely to the high cost of translation. To expand the reach of these creative works, the researchers developed an automated system using machine translation (MT) technology.

Manga presents two important problems for MT: context-awareness and multimodal translation. Format-specific challenges include the comics’ unstructured mashup of text and images and the lack of corpora or benchmarks to train and evaluate models.

The team proposes a multimodal context-aware translation framework, which they say is the first to incorporate context information obtained from an image into manga translation. The framework extracts three context types that are useful for multimodal context-aware translation: scene, reading order, and visual information.

The proposed method automatically builds a manga parallel corpus from pairs of original manga texts extracted using computer vision techniques and their translations. The researchers also created a multilingual manga dataset, effectively the first manga translation benchmark, and have made it publicly available.

Given a Japanese manga page as input, the system recognizes Japanese texts and translates them into the target language using the trained neural machine translation model. An image inpainting model meanwhile removes the original Japanese texts from the page, and the translated texts are rendered on the cleaned page with optimized font size and location.

By integrating text recognition, machine translation, and image processing into a unified system, the proposed method achieves impressive automatic image-to-image manga translation and has established a foundation for further research in this area of MT.

The paper Towards Fully Automated Manga Translation is on arXiv.

---

Reporter: Yuan Yuan | Editor: Michael Sarazen

---

Synced Report | A Survey of China’s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic — 87 Case Studies from 700+ AI Vendors

This report offers a look at how China has leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon Kindle. Along with this report, we also introduced a database covering additional 1428 artificial intelligence solutions from 12 pandemic scenarios.

Click here to find more reports from us.

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.