AI Asia China Machine Learning & Data Science Nature Language Tech Research

Meet ByteDance AI’s Xiaomingbot: World’s First Multilingual and Multimodal AI News Agent

Researchers from ByteDance AILab and Shanghai Jiao Tong University have introduced Xiaomingbot, a multilingual and multimodal news reporter.

Continuous improvements in modern natural language generation in recent years have enabled bots that can perform automatic news reporting. This has practical applications for example in minor league sports, where result data is available but it is not always cost-efficient to send human reporters to the contests. Most existing robot reporters however focus exclusively on text generation. In a bid to develop more versatile and user-friendly intelligent robot reporters, researchers from ByteDance AILab and Shanghai Jiao Tong University have introduced Xiaomingbot, a multilingual and multimodal news reporter that is able to:

  • Create news articles from input data such as scoring stats or box scores
  • Read these articles with the lifelike animation of a typical TV anchor
  • Deliver the news in multiple languages to serve global users
image.png

Xiaomingbot contains four components: a news generator, a news translator, a cross-lingual newsreader and an animated avatar. Its input is data table containing game and event records, and the output is an animated avatar reading a news article with a synthesized voice. The system targets news delivery in areas such as sports and financial reporting.

image.png
Xiaomingbot system architecture

To enable the system to generate reasonable text, a template based on table2text technology is used to write the news. The text summarization module filters and summarizes the produced news texts to provide a condensed abstract with the most important and pertinent information. Next, a state-of-the-art transformer large model is used as a machine translation module to translate the abstract into the user-specified language. A text to speech (TTS) module with multilingual capability enables Xiaomingbot to read the final script in different languages using the same voice. An animated anchor avatar with synchronized lip motion and facial expression and realistic body and clothing makes the news broadcast more natural and viewer-friendly.

image.png
User Interface of Xiaomingbot. Here, sports results are used as input and corresponding speech and visual effects are output.
image.png
Avatar animation synthesis

Xiaomingbot is a first fully functional robot reporter capable of writing, speaking, and dynamic expression to be deployed online and actively serve users. It has already written and delivered over 600,000 news articles and gained over 150,000 followers across various social media platforms.

The paper Xiaomingbot: A Multilingual Robot News Reporter is on arXiv.


Author: Hecate He | Editor: Michael Sarazen & Yuan Yuan


Synced Report | A Survey of China’s Artificial Intelligence Solutions in Response to the COVID-19 Pandemic — 87 Case Studies from 700+ AI Vendors

This report offers a look at how the Chinese government and business owners have leveraged artificial intelligence technologies in the battle against COVID-19. It is also available on Amazon Kindle.

Click here to find more reports from us.


We know you don’t want to miss any story. Subscribe to our popular Synced Global AI Weekly to get weekly AI updates.

10 comments on “Meet ByteDance AI’s Xiaomingbot: World’s First Multilingual and Multimodal AI News Agent

  1. amazing

  2. Pingback: ByteDance AI’s Xiaomingbot: World First Multilingual&Multimodal AINewsAgent – Full-Stack Feed

  3. Pingback: 史上首个3D多语言AI记者来了,不仅写作、翻译无压力,还能一线报道国际体育赛事 | 雷锋网-国内区块链-数字货币软件-网赚兼职-赢百万信息网腾讯分分组选三星计划软件,赢百万挂机软件,三星

  4. Pingback: 史上首个3D多语言AI记者来了,不仅写作、翻译无压力,还能一线报道国际体育赛事 – 聚合资讯网

  5. Pingback: 史上首個3D多語言AI記者來瞭,不僅寫作、翻譯無壓力,還能一線報道國際體育賽事 – News

  6. Pingback: 역사상 최초의 3D 다국어 AI 리포터가 여기에 있습니다. – Tech News

  7. Great blog! Do you have any suggestions for aspiring writers?
    I’m planning to start my own site soon but I’m a little lost on everything.

    Would you propose starting with a free platform like WordPress or go
    for a paid option? There are so many options out there that I’m completely confused ..
    Any recommendations? Kudos!

  8. Pingback: 史上首个3D多语言AI记者来了,不仅写作、翻译无压力,还能一线报道国际体育赛事-i格子网

  9. Pingback: The phenomenon of Toutiao - The Fix

  10. Pingback: Toutiao — Bytedance’s “secret sauce” and the future of publishing | What’s New in Publishing | Digital Publishing News

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: