What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing

kaestner@cs.cmu.edu (C. Yang, Y. Hong, G. Lewis, T. Wu, C. Kästner)
Thu, 11 Dec 2025 09:54:38 GMT

Chenyang Yang, Yining Hong, Grace Lewis, Tongshuang Wu, and Christian Kästner. What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 306--318, Los Alamitos, CA: IEEE Computer Society, November 2024.

Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including error analysis, testing, auditing, and red-teaming, to form hypotheses of what can go (or has gone) wrong with their models. To validate these hypotheses, practitioners employ data slicing to identify examples relevant to their hypotheses. However, traditional data slicing is limited by available features and programmatic slicing functions. In this work, we propose SemSlicer, a framework that supports semantic data slicing, which identifies a semantically coherent slice, without the need for existing features. SemSlicer uses Large Language Models (LLMs) to annotate datasets and generate slices from any user-defined slicing criteria. We show that SemSlicer generates accurate slices with low cost, allows flexible trade-offs between different design dimensions, reliably identifies under-performing data slices, and helps practitioners identify useful data slices that reflect systematic problems.