The rapid progress of large language models (LLMs) has greatly influenced natural language processing (NLP), driving advancements across numerous applications. However, LLM training is typically restricted to relatively short context lengths, such as 8K or 32K tokens. Extending this context length is challenging, as the memory required for storing activations and intermediate buffers grows proportionally with the context size.
In a new paper Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer, a Microsoft research team introduces the Fully Pipelined Distributed Transformer (FPDT) to address the difficulties of training long-context LLMs. This approach leverages the multiple memory hierarchies available in modern GPU clusters, enhancing hardware efficiency and cost-effectiveness while achieving exceptionally high Model FLOPs Utilization (MFU).
The team begins with a comprehensive analysis of the memory footprint associated with LLM training, identifying memory spikes in commonly used Transformer architectures. They focus on reducing redundant intermediate buffers during both the forward and backward passes.
Building on this analysis, they developed a fully pipelined distributed transformer, based on DeepSpeed Ulysses, specifically designed for LLMs with sequence lengths reaching millions of tokens. This design utilizes both GPU and host CPU memory, along with prefetching techniques, to create a near-zero overhead training process.
The researchers also introduce a double buffer system to overlap almost all prefetching with computation. This approach ensures that attention computation in the inner loop only needs to account for the latency of fetching the next query, rather than both key and value prefetching, thereby significantly reducing the GPU memory footprint.
When applied to GPT and Llama models, FPDT achieves a 16-fold increase in sequence length that can be trained on the same hardware compared to current state-of-the-art methods. Thanks to its specialized sequence chunk pipeline design, FPDT can train an 8-billion-parameter LLM with a sequence length of 2 million tokens using only 4 GPUs, while maintaining over 55% MFU. The researchers believe that their work will greatly benefit the community, enabling further exploration of LLM capabilities in long-context scenarios.
The code is available on project’s GitHub. The paper Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformer is on arXiv.
Author: Hecate He | Editor: Chain Zhang
One thing that caught my eye is the claim about “near-zero overhead training.” Does the prefetching overhead stay negligible even when you’re scaling up to millions of tokens, or does it start to bite?
Great analysis! As someone who spends hours reading tech articles, I always need a brain break. [OpenClaw](https://open-claw.me) is my go-to — a simple but fun office destruction game. Highly recommend it for stress relief between deep dives.
Wow, 16x sequence length (Video2prompt) and extreme hardware efficiency with Microsoft’s Fully Pipelined Distributed Transformer sounds incredible! Training a 2 million token LLM on just 4 GPUs with high MFU is a huge leap.
Really enjoyed reading on syncedreview.com. The practical tips are easy to apply and genuinely useful. I am working on a related project and this perspective helped me frame things more clearly. Would love to hear what methods have worked best for others here.
This is a really impressive development! Extending context length without a huge memory penalty is a game-changer for LLMs. It’s exciting to see how this will enable new applications, much like how flexible image tools can help visualize data. Perhaps it will make tasks like creating complex visual comparisons easier too, similar to what you might do with an online image editor.
That 16x sequence length boost for an 8 billion parameter LLM is impressive and makes me wonder if such efficiency helps folks using AnimateX to create anime and digital art. Design characters with AI gets way easier when the hardware isn’t a bottleneck, which is a cool thought I had while grabbing a quick snack!
Interesting read — it’s impressive to see how transformer efficiency and long-context processing keep improving. These kinds of advances are important because they make AI more practical for real user-facing applications.
I’m also building a small AI tool for manga and webtoon readers: AI Manga Translator — it uses AI to translate comic pages while preserving the original layout.