Later Is Better: Token Reduction for ViTs Under Distribution Shift

Oct 6, 2026·
Hyeongheon Cha
Hyeongheon Cha
,
Hyungjun Yoon
,
Sung-Ju Lee
· 0 min read
Abstract
This work examines how the depth at which tokens are removed affects vision-transformer robustness under distribution shift. A one-parameter power-law schedule concentrates removal in later layers, improving out-of-distribution accuracy without increasing inference cost. Its gains persist when compute is matched, even with fewer final tokens. Experiments cover different reduction methods, backbones, corruptions, additional shift benchmarks, video, vision-language tasks, and test-time adaptation methods. The findings associate early removal with errors propagating through more remaining layers and support a shared schedule without input-specific tuning.
Type
Publication
arXiv preprint