Later Is Better: Token Reduction for ViTs Under Distribution Shift
Abstract
This work examines how the depth at which tokens are removed affects vision-transformer robustness under distribution shift. A one-parameter power-law schedule concentrates removal in later layers, improving out-of-distribution accuracy without increasing inference cost. Its gains persist when compute is matched, even with fewer final tokens. Experiments cover different reduction methods, backbones, corruptions, additional shift benchmarks, video, vision-language tasks, and test-time adaptation methods. The findings associate early removal with errors propagating through more remaining layers and support a shared schedule without input-specific tuning.
Type
Publication
arXiv preprint