Visual In-Context Learning (VICL) is a prevailing way to transfer visual foundation models to new tasks by leveraging contextual information contained in in-context examples to enhance learning and prediction of query samples. The fundamental problem in VICL is how to select the best prompt to activate its power as much as possible, which is equivalent to the ranking problem of testing the in-context behavior of each candidate in the alternative set and selecting the best one. To utilize a more appropriate ranking metric and more comprehensive information among the alternative set, we propose a novel in-context example selection framework to approximately identify the global optimal prompt, i.e. choosing the best performing in-context examples from all alternatives for each query sample. Our method, dubbed Partial2Global, adopts a transformer-based list-wise ranker to provide a more comprehensive comparison within several alternatives and a consistency-aware ranking aggregator to generate globally consistent ranking. The effectiveness of Partial2Global is validated through experiments on foreground segmentation, single object detection and image colorization, demonstrating that Partial2Global selects consistently better in-context examples compared with other methods, and thus establishes the new state-of-the-arts.
Our Partial2Global involves a transformer-based list-wise ranker, which produces ranking prediction for different subsets and a onsistency-aware ranking aggregator to generate globally consistent ranking.
@article{xu2024towards,
title={Towards Global Optimal Visual In-Context Learning Prompt Selection},
author={Xu, Chengming and Liu, Chen and Wang, Yikai and Yao, Yuan and Fu, Yanwei},
journal={arXiv preprint arXiv:2405.15279},
year={2024}
}