Bridging the Gap: Induction-Head Ngram Models for Efficient, Interpretable Language Modeling

Recent large language models (LLMs) have shown impressive performance across a diverse array of tasks. However, their use in high-stakes or computationally constrained environments has highlighted the need for more interpretable and efficient alternatives. While traditional n-gram models are fully interpretable and much more computationally efficient, they typically fall short of LLMs in predictive accuracy, especially for tasks like next-token prediction.

In a new paper Interpretable Language Modeling via Induction-head Ngram Models, a research team from Microsoft Research, Seoul National University and Stanford University introduces a novel approach called Induction-head ngram models (Induction-Gram). This technique merges the interpretability and efficiency of n-gram models with insights from neural LLMs to enhance language modeling performance.

Induction-Gram is built upon Infini-Gram, an advanced, scalable n-gram model. Despite its strengths, Infini-Gram struggles with adapting to new contexts and handling queries that lack exact matches in its reference data. To address these limitations, Induction-Gram introduces a fuzzy matching mechanism that retrieves likely next-token suggestions, simulating the function of “induction heads” in transformer models.

This fuzzy matching is trained using both Cross Entropy (CE) loss and reverse Kullback-Leibler divergence (KLD) loss. Within each training batch, the researchers generate similarity pairs by sampling sequences through an LLM. The CE loss facilitates identification of the closest matches, while the reverse KLD loss fine-tunes the model to mirror similarity distributions—ensuring high similarity for close matches and lower similarity for more distant pairs.

The model employs a custom neural similarity metric, optimized to score pairs of text sequences based on the similarity of their next-token predictions. This advancement enables Induction-Gram to reach state-of-the-art next-token prediction accuracy among interpretable models.

In tests on the Pile dataset with OpenWebText as the reference, Induction-Gram boosts next-token prediction accuracy by 20 percentage points over Infini-Gram, substantially narrowing the accuracy gap between interpretable models and the black-box GPT-2.

Induction-Gram represents a significant leap forward in creating interpretable language models that approximate the performance of sophisticated LLMs, paving the way for more accessible and transparent language modeling solutions.

The code is available on project’s GitHub. The paper Interpretable Language Modeling via Induction-head Ngram Models is on arXiv.

---

Author: Hecate He | Editor: Chain Zhang

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.