Matrix multiplication (MatMul) is a fundamental operation in most neural networks, primarily because GPUs are highly optimized for these computations. Despite its critical role in deep learning, MatMul operations are a significant source of computational expense, often consuming the bulk of execution time and memory access during both training and inference phases.
In a new paper Scalable MatMul-free Language Modeling, a research team from University of California, Santa Cruz, Soochow University, University of California, Davis and LuxiTech introduces the first scalable MatMul-free language model (MatMul-free LM). Their findings demonstrate that it is possible to completely eliminate MatMul operations from large language models (LLMs) while maintaining robust performance, even at billion-parameter scales.
The MatMul-free LM achieves this by employing additive operations in dense layers and element-wise Hadamard products for self-attention-like functions. Specifically, ternary weights are used to eliminate MatMul in dense layers, similar to binary neural networks (BNNs). To remove MatMul from self-attention, the researchers optimize the Gated Recurrent Unit (GRU) to rely solely on element-wise products. This innovative model competes with state-of-the-art Transformers while eliminating all MatMul operations.
The team’s architectural perspective is inspired by Metaformer, which conceptualizes Transformers as consisting of a token mixer (for mixing temporal information, such as self-attention or Mamba) and a channel mixer (for mixing embedding/spatial information, such as feed-forward networks or Gated Linear Units (GLUs)).
To quantify the hardware benefits of their lightweight model, the researchers provide an optimized GPU implementation along with a custom FPGA accelerator. This approach reduces memory usage by up to 61% over an unoptimized baseline during training. By utilizing an optimized kernel during inference, the MatMul-free LM reduces memory consumption by more than 10× compared to unoptimized models.
To thoroughly evaluate the efficiency of their proposed architecture, the team developed a custom hardware solution on an FPGA that leverages lightweight operations beyond the capabilities of GPUs. They successfully processed billion-parameter scale models at 13 watts, achieving throughput comparable to human-readable performance, and bringing LLMs closer to brain-like efficiency.
Overall, this work not only illustrates the potential for drastically simplifying LLMs while maintaining high performance but also highlights the types of operations future accelerators should be optimized for in processing the next generation of lightweight LLMs.
The code implementation is available at project’s GitHub. The paper Scalable MatMul-free Language Modeling is on arXiv.
Author: Hecate He | Editor: Chain Zhang
We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.
Sapphire Las Vegas stands out as the premier las vegas gentlemen’s club, offering an incredible array of entertainment options. From the elegant surroundings to the top-notch service, every detail is designed to provide guests with an unforgettable night. Experience the luxury and excitement of Sapphire.
Osh University’s commitment to medical education extends to osh kyrgyzstan , ensuring our students are well-versed in this essential field of study. It’s a key component of our comprehensive medical curriculum.