Later Is Better: Token Reduction for ViTs Under Distribution Shift

Oct 6, 2026·
Hyeongheon Cha
Hyeongheon Cha
,
Hyungjun Yoon
,
Sung-Ju Lee
· 1 min read
Late-concentrated token reduction under distribution shift ( ).

Position

First author

Project Goals & Works

  1. Study how the depth of token removal affects vision-transformer robustness under distribution shift.
  2. Use a one-parameter power-law schedule to preserve tokens in early layers and concentrate removal in later layers.
  3. Match the compute budget of a uniform removal schedule without retraining or input-specific tuning.

Key Results

  1. On ImageNet-C at severity 5, improved DeiT-S with ToMe by 1.17 percentage points over uniform removal at matched compute, while using about 26% less compute than the unreduced model.
  2. Demonstrated improvements across five token-reduction methods and nine vision-transformer backbones.
  3. Combined the schedule with six test-time adaptation methods, with further evaluations on video and vision-language tasks.