A Comparison of 10 Sampling Algorithms for Configurable Systems

kaestner@cs.cmu.edu (F. Medeiros, C. Kästner, M. Ribeiro, R. Gheyi, S. Apel)
Wed, 01 Mar 2017 14:35:44 -0500

Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. A Comparison of 10 Sampling Algorithms for Configurable Systems. In Proceedings of the 38th International Conference on Software Engineering (ICSE), pages 643--654, New York, NY: ACM Press, May 2016.

Almost every software system provides configuration options to tailor the system to the target platform and application scenario. Often, this configurability renders the analysis of every individual system configuration infeasible. To address this problem, researchers proposed a diverse set of sampling algorithms. We present a comparative study of 10 state-of-the-art sampling algorithms regarding their fault-detection capability and size of sample sets. The former is important to improve software quality and the latter to reduce the time of analysis. In a nutshell, we found that the sampling algorithms with larger sample sets detected higher numbers of faults. Furthermore, we observed that the limiting assumptions made in previous work influence the number of detected faults, the size of sample sets, and the ranking of algorithms. Finally, we identified a number of technical challenges when trying to avoid the limiting assumptions, which question the practicality of certain sampling algorithms.