ICLR 2021 submission proposes LambdaNetworks, a transformer-specific method that reduces costs of modeling long-range interactions for CV and other applications.
ICLR 2021 paper An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale suggests Transformers can outperform top CNNs on CV at scale.