From Hazard Identification to Control Design: Proactive and AI-Supported Safety Engineering for ML-powered Systems

kaestner@cs.cmu.edu (Y. Hong, C. Timperley, C. Kästner)
Thu, 11 Dec 2025 09:54:38 GMT

Yining Hong, Christopher Timperley, and Christian Kästner. From Hazard Identification to Control Design: Proactive and AI-Supported Safety Engineering for ML-powered Systems. In Proceedings of the International Conference on AI Engineering - Software Engineering for AI (CAIN), pages 113--118, April 2025.

Machine learning (ML) components are increasingly integrated into software products, yet their complexity and inherent uncertainty often lead to unintended and potentially hazardous consequences, both for individuals and society at large. Despite these risks, practitioners rarely adopt proactive approaches to anticipate and mitigate potential hazards before they occur. Traditional safety engineering approaches, such as Failure Mode and Effects Analysis (FMEA) and System Theoretic Process Analysis (STPA), offer promising frameworks for systematic early risk identification but are rarely adopted. In this position paper, we argue that hazard analysis should be an integral part of developing any ML-powered software product and that greater support is needed to make this process manageable for developers. By using large language models (LLMs) to partially automate a modified STPA process with human oversight at critical steps, we expect to address two key challenges: the heavy dependency on highly experienced safety engineering experts, and the time-consuming, labor-intensive nature of traditional hazard analysis, which often impedes its integration into real-world development workflows. We illustrate our approach with a running example, demonstrating that many seemingly unanticipated issues can, in fact, be anticipated. We conclude with a call to action for the software engineering community to adopt proactive safety engineering practices for ML-powered systems.