Transformers Provably Learn Sparse Token Selection While Fully-Connected
Nets Cannot
Transformers Provably Learn Sparse Token Selection While Fully-Connected
Nets Cannot
The transformer architecture has prevailed in various deep learning settings due to its exceptional capabilities to select and compose structural information. Motivated by these capabilities, Sanford et al. proposed the sparse token selection task, in which transformers excel while fully-connected networks (FCNs) fail in the worst case. Building upon that, …