Tag: long text processing

by Synced 2024-09-09 6

Microsoft’s Fully Pipelined Distributed Transformer Processes 16x Sequence Length with Extreme Hardware Efficiency

A Microsoft research team introduces the Fully Pipelined Distributed Transformer, which leverages the multiple memory hierarchies available in modern GPU clusters, enhancing hardware efficiency and cost-effectiveness while achieving exceptionally high Model FLOPs Utilization (MFU).

by Synced 2023-10-11 1

AI Machine Learning & Data Science Research

Yale U & Google’s HyperAttention: Long-Context Attention with the Best Possible Near-Linear Time Guarantee

In a new paper HyperAttention: Long-context Attention in Near-Linear Time, a research team from Yale University and Google Research presents HyperAttention, an approximate attention mechanism not only offers practical efficiency but also delivers the best near-linear time guarantee for long contexts processing.