While large language models (LLMs) dominate the AI landscape, Small-scale Large Language Models (SLMs) are gaining traction as cost-effective and efficient alternatives for various applications. Despite their growing relevance, there is a significant gap in understanding the training behavior and computational requirements of SLMs, especially in contrast to their larger counterparts.
In a new paper Computational Bottlenecks of Training Small-scale Large Language Models, Apple researchers address this gap by conducting a systematic study of the computational bottlenecks and cost-efficiency of training SLMs, focusing on models with up to 2 billion parameters. Their work evaluates training strategies across diverse cloud infrastructure setups, offering practical insights for improving efficiency and reducing costs.
While previous research has largely concentrated on optimizing SLMs for inference, little attention has been paid to the unique challenges of their training dynamics. This oversight is significant because the computational and infrastructure demands of training LLMs often do not directly apply to SLMs. With the wide range of hardware configurations available on cloud platforms—including variations in GPU types, batch sizes, and communication protocols—it becomes essential to evaluate how these factors influence the training efficiency of SLMs. Key metrics such as loss per dollar and tokens per second provide a practical framework for this analysis.
The study aims to identify architectures that maximize performance while minimizing training costs. The researchers focus on LLaMA architectures, widely recognized in both public LLM and SLM communities. They evaluate models with 100M, 500M, 1B, and 2B parameters, conducting an extensive grid search across a range of configuration parameters to optimize for performance and cost efficiency. Each data point represents the best configuration for the specified parameters.
The configurations explored include:
- GPU Types: Three NVIDIA GPUs are evaluated—A100-40GB, A100-80GB, and H100-80GB—using BFloat16 data types for all experiments.
- GPU Numbers and Communication Protocols: Training setups range from single-node-single-GPU (1 GPU) to multi-node-multi-GPU configurations (up to 64 GPUs).
- Batch Sizes: Various sample sizes per GPU are tested to assess their impact on efficiency.
- FlashAttention: The effects of incorporating FlashAttention in the attention block are analyzed.
The researchers uncover several important insights about the computational bottlenecks and strategies for training SLMs:
- FlashAttention’s Importance: FlashAttention is more critical for SLMs than for larger LLMs, offering significant efficiency gains during training.
- Cost-Effectiveness of Hardware: High-end GPUs, such as H100-80GB and A100-80GB, do not always provide the most cost-effective solutions for SLM training.
- Optimal Distributed Training: Distributed Data Parallel (DDP) emerges as the most efficient training scheme for SLMs, outperforming alternatives.
- GPU Memory Utilization: Maximizing GPU memory usage is not necessarily the most cost-efficient strategy for training SLMs.
This research provides actionable insights for optimizing SLM training, particularly for institutions with limited resources. By highlighting the critical role of FlashAttention, evaluating the cost-effectiveness of hardware, and identifying the most efficient training schemes, the study supports broader adoption and optimization of SLMs in low-resource environments.
Apple’s systematic evaluation of SLM training dynamics sheds light on the computational and cost-efficiency challenges unique to these models. By addressing these bottlenecks, the study paves the way for the more efficient development of SLMs, ensuring that high-quality AI tools remain accessible to a wider range of researchers and organizations. These findings are poised to advance the adoption of SLMs as a viable alternative to LLMs in cost-sensitive and resource-constrained scenarios.
The paper Computational Bottlenecks of Training Small-scale Large Language Models is on arXiv.
Author: Hecate He | Editor: Chain Zhang