Machine Learning & Data Science Research

Japanese Manga Translation Via Multimodal Context-Aware Framework

A new machine translation method enables global manga fans to enjoy immediate translations of their favourite Japanese comics.

A new machine translation method from Japanese start-up Mantra Inc., Yahoo Japan and the University of Tokyo enables global manga fans to enjoy immediate translations of their favourite Japanese comics. The researchers say this is the first comprehensive system for fully automated manga translation from the original Japanese into English or Chinese.

Comics are an integral component of many cultures. US superhero sagas, Korean webtoons and Chinese manhua all have unique styles and characteristics. Most comics however remain readily available only in their domestic markets, due largely to the high cost of translation. To expand the reach of these creative works, the researchers developed an automated system using machine translation (MT) technology.

Manga presents two important problems for MT: context-awareness and multimodal translation. Format-specific challenges include the comics’ unstructured mashup of text and images and the lack of corpora or benchmarks to train and evaluate models.

The team proposes a multimodal context-aware translation framework, which they say is the first to incorporate context information obtained from an image into manga translation. The framework extracts three context types that are useful for multimodal context-aware translation: scene, reading order, and visual information.

Proposed manga translation framework

The proposed method automatically builds a manga parallel corpus from pairs of original manga texts extracted using computer vision techniques and their translations. The researchers also created a multilingual manga dataset, effectively the first manga translation benchmark, and have made it publicly available.

Proposed framework of parallel corpus construction

Given a Japanese manga page as input, the system recognizes Japanese texts and translates them into the target language using the trained neural machine translation model. An image inpainting model meanwhile removes the original Japanese texts from the page, and the translated texts are rendered on the cleaned page with optimized font size and location.

Results of fully automatic manga translation from Japanese to English and Chinese

By integrating text recognition, machine translation, and image processing into a unified system, the proposed method achieves impressive automatic image-to-image manga translation and has established a foundation for further research in this area of MT.

The paper Towards Fully Automated Manga Translation is on arXiv.

Reporter: Yuan Yuan | Editor: Michael Sarazen


Synced Report | A Survey of China’s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic — 87 Case Studies from 700+ AI Vendors

This report offers a look at how China has leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon KindleAlong with this report, we also introduced a database covering additional 1428 artificial intelligence solutions from 12 pandemic scenarios.

Click here to find more reports from us.

AI Weekly.png

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.

4 comments on “Japanese Manga Translation Via Multimodal Context-Aware Framework

  1. Pingback: [R] Japanese Manga Translation Via Multimodal Context-Aware Framework –

  2. Pingback: [R] Japanese Manga Translation Via Multimodal Context-Aware Framework – ONEO AI

  3. I am actually highly obliged to express my very thankfulness to you for your very knowledge-sharing efforts that you are actually doing it for all the people you every time just see around yourself. Thank you!

  4. I want i join and read your translate

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: