The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Products

kaestner@cs.cmu.edu (N. Nahar, H. Zhang, G. Lewis, S. Zhou, C. Kästner)
Thu, 11 Dec 2025 09:54:38 GMT

Nadia Nahar, Haoran Zhang, Grace Lewis, Shurui Zhou, and Christian Kästner. The Product Beyond the Model -- An Empirical Study of Repositories of Open-Source ML Products. In Proceedings of the 47th International Conference on Software Engineering (ICSE), pages 1540--1552, April 2025.

Machine learning (ML) components are increasingly incorporated into software products for end-users, but developers face challenges in transitioning from ML prototypes to products. Academics have limited access to the source of commercial ML products, challenging research progress. In this study, first, we contribute a novel process to identify 262 open-source ML products among more than half a million ML-related projects on GitHub. Then, we qualitatively and quantitatively analyze 30 open-source ML products to answer six broad research questions about development practices and system architecture. We find that the majority of the ML products in our sample represent startup-style development reported in past interview studies. We report 21 findings, including limited involvement of data scientists in many ML products, unusually low modularity between ML and non-ML code, diverse architectural choices on incorporating models into products, and limited prevalence of industry best practices such as model testing, pipeline automation, and monitoring. Additionally, we discuss 7 implications of this study on research, development, and education, including the need for tools to assist teams without data scientists, education opportunities, and open-source-specific research for privacy-preserving telemetry.