(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs

kaestner@cs.cmu.edu (W. Ma, C. Yang, C. Kästner)
Tue, 2 Jul 2024 17:25:21 GMT

Wanqin Ma, Chenyang Yang, and Christian Kästner. (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs. In Proceedings of the International Conference on AI Engineering - Software Engineering for AI (CAIN), pages 166--171, April 2024.

Large Language Models (LLMs) are increasingly integrated into software applications. Downstream application developers often access LLMs through APIs provided as a service. However, LLM APIs are often updated silently and scheduled to be deprecated, forcing users to continuously adapt to evolving models. This can cause performance regression and affect prompt design choices, as evidenced by our case study on toxicity detection. Based on our case study, we emphasize the need for and re-examine the concept of regression testing for evolving LLM APIs. We argue that regression testing LLMs requires fundamental changes to traditional testing approaches, due to different correctness notions, prompting brittleness, and non-determinism in LLM APIs.