efficient inference

MixA: A Mixed Attention approach with Stable Lightweight Linear Attention to enhance Efficiency of Vision Transformers at the Edge

We propose MixA, a mixed attention approach for improving ViT efficiency on edge devices while preserving competitive accuracy. By keeping ReLU-based quadratic attention in critical layers and replacing less critical ones with the proposed SteLLA linear attention module, MixA delivers substantial speedups with minimal performance loss.

Sabbir Ahmed, Jingtao Li, Weiming Zhuang, Chen Chen, Lingjuan Lyu