vision transformer

We propose MixA, a mixed attention approach for improving ViT efficiency on edge devices while preserving competitive accuracy. By keeping ReLU-based quadratic attention in critical layers and replacing less critical ones with the proposed SteLLA linear attention module, MixA delivers substantial speedups with minimal performance loss.