Towards Verifiably Safe Tool Use for LLM Agents

kaestner@cs.cmu.edu (A. Doshi, Y. Hong, C. Xu, E. Kang, A. Kapravelos, C. Kästner)
Mon, 13 Jul 2026 11:18:49 GMT

Aarya Doshi, Yining Hong, Congying Xu, Eunsuk Kang, Alexandros Kapravelos, and Christian Kästner. Towards Verifiably Safe Tool Use for LLM Agents. In Proceedings of the Proc. International Conference on Software Engineering -- New Ideas Track (ICSE-NIER), pages 201--205, New York, NY: ACM Press, April 2026.

Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents. While this empowers agents, unintended tool interactions may also introduce risks, such as leaking sensitive data or overwriting critical records, which are unacceptable in enterprise contexts. Current approaches, such as model-based safeguards, enhance reliability but cannot guarantee system safety. Methods like information flow control (IFC) and temporal constraints aim to provide guarantees but often require extensive human annotation. We propose a process that starts with applying System-Theoretic Process Analysis (STPA) to identify hazards in agent workflows, derive safety requirements, and formalize them as enforceable specifications on data flows and tool sequences. To enable this, we introduce a capability-enhanced Model Context Protocol (MCP) framework that requires structured labels on capabilities, confidentiality, and trust level. Together, these contributions aim to shift safety from ad hoc reliability fixes to proactive guardrails with guarantees, while reducing dependence on user confirmation and making autonomy a deliberate design choice.