Matrix Multiplication-Free Language Models Maintain Top-Tier Performance at Billion-Parameter Scales
In a new paper Scalable MatMul-free Language Modeling, a research team introduces the first scalable MatMul-free language model, demonstrating that it is possible to completely eliminate MatMul operations from large language models (LLMs) while maintaining robust performance, even at billion-parameter scales.