The paradigm of pretraining a backbone on a large set of (often unlabeled) images has gained popularity.
The quality of the resulting features is commonly measured by freezing the backbone and training different task heads on top of it.
However, current evaluations cover only classifications of whole images or require complex dense task heads which introduce a large number of parameters and add their own inductive biases.
In this work, we propose dense attentive probing, a parameter-efficient readout method for dense prediction on arbitrary backbones -- independent of the size and resolution of their feature volume.
To this end, we extend cross-attention with distance-based masks of learnable sizes.
We employ this method to evaluate 18 common backbones on dense predictions tasks in three dimensions: instance awareness, local semantics and spatial understanding.
We find that DINOv2 outperforms all other backbones tested -- including those supervised with masks and language -- across all three task categories.
Furthermore, our analysis suggests that self-supervised pretraining tends to yield features that separate object instances better than vision-language models.
Results
We train Dense Attentive Probing as a readout on top of several frozen backbones and evaluate downstream task performances for dense prediction tasks. Due to its small number of parameters, we can measure dense prediction performance directly, similar to linear probing for tasks that involve the whole image. These are our results:
Here we specifically focus on the instance awareness tasks, i.e. we assess how well individual instances are encoded in the backbone features. Specifically we consider the tasks object detection, instance boundary prediction and instance discrimination.
Performance on instance-awareness tasks. Self-supervised methods outperform vision-language methods.
Backbone property correlations
Using the data from above, we analyze to which degree performance is related to general properties of the backbone (image size, feature dimension, number of parameters), independent of the training method.
Correlation of task performance with backbone properties (p-values in brackets).
Semantic segmentation and depth estimation improve with larger backbone size (in terms of parameters) and higher input resolution, showing significant positive correlations.
Semantic segmentation also correlates slightly with spatial feature size.
Object detection shows weaker correlations with all three backbone variables, suggesting less sensitivity to these factors.
Speed-accuracy trade-off
In practice, inference speed and memory requirements often matter. If these are constrained, a trade-off must be found. The following figure helps to find the right backbone depending on task and inference compute.
Speed-accuracy trade-off in object detection, depth estimation and semantic segmentation.
Method: Dense Attentive Probing
The idea of Dense Attentive Probing is to use cross-attention as a minimal task head on top of frozen backbones. Frozen means that no weights can be changed. Due to the small number of parameters, we directly assess the quality of the backbone's features for the respective task.
General architecture: We feed the features from frozen backbones into the DEAP adapter which consists of two cross attention layers followed by deconvolutions.
Our method generates outputs for different tasks using a single cross-attention layers that attend to the feature volume and whose queries are responsible for generating a local region of the output.
We enable the queries to focus on local regions in the feature volume (i.e. neighborhoods in the feature volume close to the queries location) through masking the attention.
The attention mask depends on the query location and a learnable parameter σ.
Dynamic Output Size
It is possible to generate dense predictions at different resolutions than the training resolution. In some cases, more details become visible this way.
The output size can be changed after training.
Method Comparison
How about other methods for dense probing? We compare to three baselines: Bicubic upscaling, a CNN readout and a FeatUp-based upscaler.
Comparison of our method with a CNN baseline.
For more information, see our Paper and the corresponding source code.
Citation
@article{
lueddecke2025deap,
title={Characterizing Vision Backbones for Dense Prediction with Dense Attentive Probing},
author={Timo Lüddecke and Alexander Ecker},
journal={Transactions on Machine Learning Research},
year={2025},
url={https://openreview.net/forum?id=neMAx4uBlh},
note={accepted}
}
Acknowledgements
This publication was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - Project-ID 454648639 - SFB 1528
