Integrating and Testing the Literature: A Conceptual Replication of Continuous Integration Pain Points

kaestner@cs.cmu.edu (D. Widder, M. Hilton, C. Kästner, B. Vasilescu)
Thu, 19 Dec 2019 17:21:43 -0500

David Widder, Michael Hilton, Christian Kästner, and Bogdan Vasilescu. Integrating and Testing the Literature: A Conceptual Replication of Continuous Integration Pain Points. In Proceedings of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 647--658, New York, NY: ACM Press, August 2019.

Continuous integration (CI) is an established software quality assurance practice, and the focus of much prior research with a diverse range of methods and populations. In this paper, we conduct a literature review of 37 papers on CI pain points. We then conduct a conceptual replication study on results from these papers using a triangulation design consisting of a survey with 132 responses, 12 interviews, and two logistic regressions predicting CI abandonment and switching on a dataset of 6,239 GitHub projects. We report and discuss which past results we were able to replicate, those for which we found conflicting evidence, those for which we did not find evidence, and the implications of these cases on future CI research, CI tool builders, and CI users.