You Can't Remove Humans From Software Engineering

There's no doubt that artificial intelligence is transforming how software is built, and tools that can generate code in seconds have sparked excitement about the future of programming. But they're also raising questions about trust, accountability, the role of human expertise and the future of software engineering education.

For Claire Le Goues, a professor in Carnegie Mellon University's School of Computer Science, those questions sit at the heart of software engineering. Her research program in the Software and Societal Systems Department explores how people can build confidence in increasingly complex software systems, especially as AI becomes more integral to their creation. While advances in AI may make programming more accessible and automate some of the more tedious aspects of software development, Le Goues argues that the field's biggest challenges remain deeply human: defining goals, understanding intent and determining whether a system is actually doing what people want it to do.

Most people don't think much about software engineering. Why should they care about research on trustworthy software?
Software touches nearly every part of modern life, from healthcare and finance to everyday things like the Spotify app on your phone. But the challenge is that software is incredibly difficult to get completely right. In practice, no software is perfect. Instead, we use a variety of techniques to build confidence that it will behave the way we expect. I study the kinds of evidence we need to trust software that was written, at least in part, by AI. That's ultimately what my research is about.

AI can quickly generate software. How does that change the way we think about trust, correctness and reliability?
Entire areas of computer science are devoted to mathematically proving what software will or won't do under certain conditions. But I tend to think about correctness a little differently. Software exists to solve human problems, and whether it is "correct" often depends on whether it meets human expectations. In that sense, trust and reliability are as much social and economic questions as technical ones. AI doesn't eliminate those questions.

A lot of your work explores the relationship between human intent and software. Why is understanding what people want often harder than writing the code itself?
Because software is ultimately defined with respect to human goals. The hardest part of software engineering has never been typing code into a computer. It's figuring out what problem you're trying to solve and how you'll know when you've solved it. People often describe their goals in ways that are ambiguous, incomplete or dependent on context. Before we can verify that software is behaving correctly, we need to understand what "correct" means. But how can we write the exact mathematical representation of human intention?

AI presents an opportunity because it may help connect pieces of the software development process that have traditionally been separate. We already have techniques for generating code and increasingly powerful methods for proving properties about code. The difficult part, again, is creating a formal representation of what a person actually wants. A lot of existing verification work assumes that someone has already written down those requirements in a precise mathematical form. But determining how to express human intent and model the world that software operates in is still an open problem. AI may help with that translation process, but it's also where many of the hardest challenges remain.

As software systems become more automated and complex, how do we ensure humans can understand, trust and take responsibility for them?
I don't think you can remove humans from software engineering. It's fundamentally about solving human problems. Even if AI helps write code, humans still have to decide what problems are worth solving, what success looks like and whether a system behaves appropriately. At the same time, there is a real concern that some AI-generated systems can be more opaque than traditional software. Historically, trust in software has come from a collective social process. People write code, review it, test it, discuss it and build shared confidence in it. As systems become more automated, we may need new forms of evidence and new ways of establishing trust, but the human role doesn't disappear.

And despite concerns that AI will replace programmers, I don't think the underlying problem has changed. Throughout the history of computing, we've continually developed better tools that raise the level of abstraction and make programming easier. AI is another step in that progression. It may make some kinds of software development dramatically easier — especially for smaller applications, — but large, complex systems will still require engineering expertise. The challenge shifts, but it doesn't go away.

As AI becomes more capable of writing code, what does that mean for the value of a computer science education?
I don't think the value of computer science disappears simply because writing code becomes easier. Programming languages themselves were created so people wouldn't have to write machine code. AI is another step in that progression. It's not exactly the same kind of technology as a new programming language, and new challenges come with it, but the idea of tools changing how we program isn't new.

What's interesting is that many people come to computer science because they're interested in solving problems in another domain. They want to build tools for healthcare, education, science, art or business. If AI lowers the barriers to creating software, that could allow more people to apply computing to the things they care about.

At the same time, expertise definitely still matters. Understanding how software works, how to reason about its behavior, and how to determine whether it can be trusted remains important. The skills may evolve, but the need for people who can think critically about systems, requirements and evidence isn't going away.

What makes you optimistic about the future of trustworthy software engineering and the role AI might play in it?
Part of it is that I find software to be genuinely fascinating. Even when you understand the principles behind it, there are still moments where it feels almost magical. More practically, I'm optimistic because AI has the potential to automate some of the tedious parts of software development while helping people focus on higher-level reasoning and problem solving.

I'm also optimistic about the specific opportunities AI creates for trust and verification. There are concrete research directions that could make long-standing verification techniques easier to use and more broadly applicable. Many of these approaches have historically been too difficult or expensive to deploy outside niche domains. If AI can lower those barriers, we may be able to build stronger evidence about software behavior across a much wider range of systems. That won't solve every problem, but it could help us build software that is both more capable and more trustworthy in all realms.