Learning to Sample: Exploiting Similarities Across Environments to Learn Performance Models for Configurable Systems

kaestner@cs.cmu.edu (P. Jamshidi, M. Velez, C. Kästner, N. Siegmund)
Wed, 28 Nov 2018 13:18:40 -0500

Pooyan Jamshidi, Miguel Velez, Christian Kästner, and Norbert Siegmund. Learning to Sample: Exploiting Similarities Across Environments to Learn Performance Models for Configurable Systems. In Proceedings of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 71--82, New York, NY: ACM Press, November 2018.

Most software systems provide options that allow users to tailor the system in terms of functionality and qualities. The increased flexibility raises challenges for understanding the configuration space and the effects of options and their interactions on performance and other non-functional properties. To identify how options and interactions affect the performance of a system, several sampling and learning strategies have been recently proposed. However, existing approaches usually assume a fixed environment (hardware, workload, version) such that learning has to be repeated when the environment changes. Repeating learning and measurement for each environment is expensive and often practically infeasible. Instead, we pursue a strategy that transfers knowledge across environments, but sidesteps heavyweight and expensive transfer-learning strategies. Based on empirical insights about common relationships regarding (i) influential options, (ii) their interactions, and (iii) their performance distributions, our approach L2S (Learning to Sample) selects better samples in the target environment based on information from the source environment. It progressively shrinks the configuration space and adaptively concentrates on interesting regions of the configuration space. With both synthetic benchmarks and several real systems, we demonstrate that L2S outperforms state of the art performance learning and transfer-learning approaches in terms of measurement effort and learning accuracy.