Scalable Analysis of Variable Software

kaestner@cs.cmu.edu (J. Liebig, A. von Rhein, C. Kästner, S. Apel, J. Dörre, C. Lengauer)
Sat, 05 Mar 2016 18:41:39 -0500

Jörg Liebig, Alexander von Rhein, Christian Kästner, Sven Apel, Jens Dörre, and Christian Lengauer. Scalable Analysis of Variable Software. In Proceedings of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 81--91, New York, NY: ACM Press, August 2013.

The advent of proper variability management and generator technology enables users to derive individual variants from a variable code base solely based on a selection of desired configuration options. This approach gives rise to a huge configuration space, but the high degree of variability comes at a cost: classic analysis methods do not scale any more; there are simply too many potential variants to analyze. To address this issue, researchers and practitioners usually apply sampling techniques—only a subset of all possible variants is analyzed. While sampling promises to reduce the analysis effort significantly, the information obtained is necessarily incomplete. Furthermore, it is unknown whether sampling strategies scale to billions of variants, because even samples may be huge and expensive to compute. Recently, researchers have begun to develop variability-aware analyses that analyze the variable code base directly with the goal to exploit the similarities among individual variants to reduce analysis effort. However, while being promising, so far, variability-aware analyses have been applied mostly only to small academic systems. To learn about the mutual strengths and weaknesses of variability-aware and sampling-based analyses of large-scale, real-world software systems, we compared the two by means of two concrete analysis implementations (type checking and liveness analysis) applied to three subject systems: the Busybox tool suite, the x86 Linux kernel, and the cryptographic library OpenSSL. A key result is that in these settings already setting up sampling techniques is challenging while variability-aware analysis even outperforms most sampling approximations with respect to analysis time.