Tag: Transformers

AI Research

Transformers Scale to Long Sequences With Linear Complexity Via Nyström-Based Self-Attention Approximation

Researchers from the University of Wisconsin-Madison, UC Berkeley, Google Brain and American Family Insurance propose Nyströmformer, an adaption of the Nystrom method that approximates standard self-attention with O(n) complexity.