Attention Mechanisms

by Synced 2021-08-30 4

Tsinghua U & Microsoft Propose Fastformer: An Additive Attention Based Transformer With Linear Complexity

A team from Tsinghua University and Microsoft Research Asia proposes Fastformer, an efficient Transformer variant based on additive attention that achieves effective context modelling with linear complexity.

by Synced 2021-08-05 3

AI Machine Learning & Data Science Nature Language Tech Research

Google’s H-Transformer-1D: Fast One-Dimensional Hierarchical Attention With Linear Complexity for Long Sequence Processing

A Google Research team draws inspiration from two numerical analysis methods — Hierarchical Matrix (H-Matrix) and Multigrid — to address the quadratic complexity problem of attention mechanisms in transformer architectures, proposing a hierarchical attention scheme that has linear complexity in run time and memory.

by Synced 2020-08-22 2

Machine Learning & Data Science Nature Language Tech Popular

Sepp Hochreiter on Parallels Between Attention Mechanisms and Modern Hopfield Networks

Researchers argue that “attention mechanism is the update rule of a modern Hopfield network with continuous states.”