Later Is Better: Token Reduction for ViTs Under Distribution Shift
Oct 6, 2026·
,,·
1 min read
Hyeongheon Cha
Hyungjun Yoon
Sung-Ju Lee
Late-concentrated token reduction under distribution shift (
).Position
First author
Project Goals & Works
- Study how the depth of token removal affects vision-transformer robustness under distribution shift.
- Use a one-parameter power-law schedule to preserve tokens in early layers and concentrate removal in later layers.
- Match the compute budget of a uniform removal schedule without retraining or input-specific tuning.
Key Results
- On ImageNet-C at severity 5, improved DeiT-S with ToMe by 1.17 percentage points over uniform removal at matched compute, while using about 26% less compute than the unreduced model.
- Demonstrated improvements across five token-reduction methods and nine vision-transformer backbones.
- Combined the schedule with six test-time adaptation methods, with further evaluations on video and vision-language tasks.