Stanford U Demonstrates Meta-Reinforcement Agents Gain Language Skills Without Direct Language Supervision

Despites that Large Language Models (LLMs) become the status quo in Natural Language Processing community, most of the existing state-of-the-art models need to be trained directly on language tasks to obtain language skills. On the contrary, humans can indirectly lean language as a byproduct of achieving non-language objectives.

Motivated by this observation, a question arise: Can embodied reinforcement learning agents gain language skill in a similarly indirectly manner? In a new paper Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learning, a Stanford University research team investigates this question in their customized multi-task environment and affirms that simple language skills can emerge in meta-RL agents without direct language supervision.

The main focus of this work is to investigate whether RL agents can learn language indirectly without any language supervision. To this end, the team first designs an office navigation environment where to goal is to find a target office as soon as possible.

In their exploration in their customized office environment, the team summarizes that they aim to answer the following four questions:

1. Our main question: Can agents learn language without explicit language supervision?
2. Can agents learn to read other modalities beyond language, such as a pictorial map?
3. What factors impact language emergence?
4. Do these results scale to 3D environments with high dimensional pixel observations?

To find out whether language can emerge, the team first trained DREAM on the 2D office with language floor plans. They observed that DREAM is able to learn an exploration policy that navigates to and reads the floor plan, then the agent leverages these information to walk to the goal office room, achieving near-optimal returns. Moreover, it can generalize to unseen relative step counts and to new layouts as well as probing the learned representation of the floor plan.

They further trained DREAM on the 2D variant of the office with pictorial floor plans, and its success to walk to the target office prove that it can also read other modalities. Next, they demonstrate that learning algorithm, amount of meta-training data and size of the model have impacts on the emergence of learning.

Finally, they scale the office environment to 3D domain and shows that DREAM is also able to read the floor plan and solves the tasks without direct language supervision.

Overall, this work verifies that language can emerge as a byproduct of solving non-language tasks in meta-RL agents.

The paper Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learning on arXiv.

---

Author: Hecate He | Editor: Chain Zhang

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.