NVIDIA’s Wolf: World Summarization Framework Beats GPT-4V on Video Captioning by 55.6%

Video captioning is essential for enhancing content accessibility and searchability by providing precise and searchable descriptions of video content. However, the task of generating accurate, descriptive, and detailed video captions remains challenging due to several factors: the limited availability of high-quality labeled data and the additional complexity involved in video captioning, such as temporal correlations and camera motion, which are not present in image captioning.

In response to these challenges, in a new paper Wolf: Captioning Everything with a World Summarization Framework, a collaborative research team from NVIDIA, UC Berkeley, MIT, UT Austin, the University of Toronto, and Stanford University has developed a novel approach known as the WOrLd summarization Framework (Wolf). This automated captioning framework significantly advances video captioning by enhancing CapScore—both in terms of quality (improved by 55.6%) and similarity (improved by 77.4%)—compared to GPT-4V.

The research team highlights three key contributions of their work:

• Introduction of Wolf: They have developed the first world summarization framework for video captioning, alongside a new LLM-based metric called CapScore for evaluating caption quality. Their method demonstrates a substantial improvement in CapScore.
• Creation of the Wolf Benchmark: The team introduces the Wolf benchmark, which includes four human-annotated datasets covering autonomous driving, general scenes from Pexels, and robotics videos. These datasets, along with human-annotated captions, form the Wolf Dataset.
• Open-source Initiative: The code, data, and leaderboard associated with Wolf will be open-sourced and maintained on the Wolf webpage, with ongoing efforts to enhance the Wolf Dataset, codebase, and CapScore.

Wolf employs a sophisticated approach, blending expert models to generate comprehensive and precise video captions. The framework leverages both image-level and video-level models to produce diverse and detailed captions, which are then cross-validated through summarization. Specifically, Wolf uses CogAgent and GPT-4V for generating image-level captions, and VILA-1.5 and Gemini-Pro-1.5 for video-level captions.

Chain-of-thought Summarization in Image-level Models. Given that image-based Visual Language Models (VLMs) are trained on more extensive datasets than video-based VLMs, the process begins with image-based VLMs to generate captions. The video is divided into sequential images, with key frames sampled at two per second. Image 1 is processed by the Image-level Model to generate Caption 1, which includes detailed scene-level information and object locations. Considering the temporal relationship between frames, both Caption 1 and Image 2 are then used to generate Caption 2. This process continues until captions are generated for all key frames. The information from these captions is then summarized by GPT-4 using a prompt designed to ensure accurate temporal representation in the video description.

LLM-based Video Summarization. Following the generation of image-level captions, these are summarized into a single cohesive caption. This process involves prompts that summarize both visual and narrative elements from the image and video model descriptions, with particular attention to motion behavior.

The empirical results show that Wolf outperforms existing state-of-the-art solutions, including both research (VILA-1.5, CogAgent) and commercial (Gemini-Pro-1.5, GPT-4V) tools. The research team hopes that Wolf will set a new benchmark in video captioning quality, raise awareness in the field, and foster further developments within the community.

The code is available on project’s webpage. The paper Wolf: Captioning Everything with a World Summarization Framework is on arXiv.

---

Author: Hecate He | Editor: Chain Zhang