A recurring theme in mathematical software evaluation is the generalization of rankings of algorithms on test problems to build knowledge-based recommender systems for algorithm selection. A key issue is to profile algorithms in terms of the qualitative characteristics of benchmark problems. In this brief methodological note, we adapt a novel all-pairs algorithm for the profiling task: Given performance rankings for m algorithms on n problem instances each described with p features, identify a (minimal) subset of p that is useful for assessing the selective superiority of all combinations of algorithms. We show how techniques presented in the mathematical software literature are inadequate for such profiling purposes. In conclusion, we also address various statistical issues underlying the effective application of this technique.

full paper