IBM Leverages Reinforcement Learning to Achieve SOTA Performance on Text and Knowledge Base Generation

Knowledge bases (KBs) can be used to store complex structured and unstructured information, and are a powerful tool for capturing real-world information with complex relationships. Automatic KB generation from free-form text and the generation of semantically meaningful text from KBs are crucial and challenging research areas in machine learning.

In the new paper ReGen: Reinforcement Learning for Text and Knowledge Base Generation Using Pretrained Language Models, an IBM research team proposes ReGen, a method for bidirectional generation of text and graph that leverages reinforcement learning (RL) to push the performance of text-to-graph (T2G) and graph-to-text (G2T) generation tasks to a higher level.

The team summarizes their main contributions as:

1. Propose using RL-based sequence training — specifically SCST — for both G2T and T2G tasks. This is the first time that RL-based training has been proposed for the bi-directional generation of text and graphs.
2. Demonstrate that our approach provides better performance than the best systems reported for the WebNLG 2020+ Challenge.
3. Provide a thorough investigation of SCST-based training for both T2G and G2T tasks, including best rewards combination.
4. Construct subject and relation-object boundaries from TEKGEN sentence-triples pairs and show the performance of our approach for both T2G and G2T tasks.
5. Adapt the large-scale TEKGEN corpus (Agarwal et al., 2021) for T2G and G2T tasks and confirm the benefit of the SCST-based fine-tuning approach over CE-trained baselines.

Inspired by the idea that sequence generation tasks can be re-framed such that a model chooses the best word within a given vocabulary, the team reformulates Seq2Seq generation into the RL framework. Simply put, an agent will define a policy that selects each word during the generation procedure. In this way, the model is employed as its own critic, enabling self-critical sequence training (SCST).

The team used T5 PLMs – t5-large (770M parameters) and t5-base (220M parameters) for their evaluation experiments. The models were fine-tuned to be either specialized on T2G (MT) or G2T (MG) tasks, or to accommodate both generation directions (MT+G).

The team reported results on the WebNLG+ 2020 (v3.0) datasets in the WebNLG 2020 Challenge, which comprises two tasks: RDF-to-text generation (G2T) and text-to-RDF semantic parsing (T2G). They also used the TEKGEN dataset to test the robustness of their proposed system, using BLEU, BLEU_NLTK, METEOR, and chrF++ as evaluation metrics.

The results demonstrate that the use of RL via SCST benefits graph and text generation on WebNLG+ 2020 and TEKGEN datasets, significantly improving on previously published results to achieve state-of-the-art performance on WebNLG+ 2020 for both text-to-graph and graph-to-text generation tasks.

In their future work, the researchers plan to develop an SCST variant that better leverages the unique structure of graphs — either by performing more sensible graph-dependent sampling or by investigating alternative reward schemes more attuned to the integration of the content and structure of graphs.

The paper ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models is on arXiv.

---

Author: Hecate He | Editor: Michael Sarazen, Chain Zhang

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.