scaling law | Synced

by Synced 2023-12-31 3

Breaking LLMs’ Limits: Upstage AI’s SOLAR 10.7B Shines Bright with Simple Scaling Magic

In a new paper SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling, a Upstage AI research team introduces depth up-scaling (DUS), which emerges as an efficient and uncomplicated technique for amplifying LLMs, surpassing existing open-source state-of-the-art LLMs, such as Llama 2 and Mistral 7B.

by Synced 2022-11-28 0

AI Machine Learning & Data Science Research

NeurIPS 2022 | Meta AI, Stanford & Tübingen U Beat Neural Scaling Laws via Data Pruning

In the NeurIPS 2022 Outstanding Paper Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning, a research team from Stanford University, University of Tübingen and Meta AI demonstrates in theory and practice how data pruning techniques can break beyond the power law scaling of error versus dataset size.

by Synced 2022-07-27 3

AI Machine Learning & Data Science Research

Google & DeepMind Study the Interactions Between Scaling Laws and Neural Network Architectures

In the new paper Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?, a research team from Google and DeepMind posits that understanding the connections between neural network architectures and scaling laws is essential for designing and evaluating new models. The team pretrains and finetunes over 100 models to reveal useful insights on the scaling behaviours of ten diverse model architectures.