HW9: Tests as Guardrails

CMU 15-114: Effective Coding with AI

Due Date: Sun. 11/15 at 8:00 PM
Time Estimate: ~5 hours
Deliverables: Your project repo with a working test suite and CI, a TESTING.md, and a prompt log, plus a Google form submission
You will be working in code you already wrote, most likely your Project 2. There is no new app to build here. If you would rather keep your Project 2 submission frozen exactly as it was graded, fork it or branch it and work there. (Other valid starting points are listed at the end of the overview.)
You will need Node installed (version 22 or newer; 26 is the current LTS, and it's what CI will run). Nothing else to install: the test runner is built in.

Assignment Overview

The premise: You have written a lot of code this semester and you have likely never written any tests. Typically we've checked whether things worked by opening the page and clicking around, which becomes insufficient when the project becomes large or critically important, or when authored by many people.

In this assignment you build a test suite to tell you whether your code still works without you having to look. This test suite will run automatically on every push.

This likely requires refactoring: Most of your Project 2 code probably cannot be tested in its current form, because the logic is tangled up with the buttons and the page. You'll begin by separating this logic. Note that code that is hard to test is almost always indicative of code that could be more cleanly organized.

Relating this to prior assignments: In HW7 you wrote acceptance criteria in English. In HW8 you built a checker skill that read your code and reported on it. Both of those depend on something reading carefully and telling you the truth. A test doesn't read anything and has no opinion. It runs, and it either passes or it fails.

Learning Objectives:

⚠️ You do not have to use Project 2. Anything you built in this class can serve as the base, and so can a codebase we wrote for you. Pick whichever of these fits your situation:

1. Your Project 2. This is likely the best choice if there is logic in it worth testing. You already understand the code, and the tests you write will keep being useful to you.
2. An earlier assignment from this class. HW4 and HW6 are strong candidates, since both have real logic sitting behind the interface. The games from HW2 and HW7 work well too, because collision checks and scoring rules are unusually easy to test. Project 1 is a poor fit, since a portfolio page has almost no logic in it.
3. The example project we provide. A small course scheduling app with no backend. It is here for you if you joined the course partway through and have no Project 2, or if you have looked honestly at your own work and cannot find anything in it worth testing.

Every requirement below applies identically whichever one you choose, and your grade does not depend on which you picked. Just name your starting point in the first line of your TESTING.md. If you are not sure which situation you are in, ask us before you invest lots of time.

Why This Matters Now

Tests have been good practice for about fifty years, and for most of that time the argument for them was that they catch your own mistakes. That argument is still true (if perhaps a little boring).

Recently, the dominant pattern in professional AI-assisted work has become the use of long-running agents: you hand them a task, they work for a while, they run some tests to check what fails, and they repeat when the tests pass. The industry benchmark for this kind of work, SWE-bench, doesn't ask the model whether it finished. It runs the repository's test suite and believes that instead.

That loop is the whole reason you can leave an agent alone. Without it, an agent's only source of truth about its own work is its own judgment, which is roughly like proofreading your own work at 3am. A test suite provides you with confidence and can indicate to the agent when it is done working.

Two common failure points

Tests sometimes don't test anything. On request, AI will generate a lot of tests quickly, and many will assert much less than they appear to. A test that only confirms a function returned something, or that it did not throw, will pass no matter how wrong the answer is.∗ The number of tests is not directly indicative of the quality of the test suite. This is why one of the requirements below is to break your own code on purpose.

Agents sometimes attack the test instead of the bug. When an agent is told to make the tests pass and repairing the code is hard, weakening the assertion or deleting the test file can appear to be a valid strategy, and it is much easier. This is well documented and you should expect to see it. It is not malice; it is the agent optimizing exactly what you asked for, which is a lesson that generalizes well past testing.

∗ A 2026 study had three agentic IDEs (Kiro, Antigravity, and Cursor, each with Claude Sonnet 4.5 as the underlying agent) generate unit tests for fifteen student-developed Python and JavaScript/TypeScript repositories, and found "a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage." Weak assertions and missing edge cases turned up more often than outright failures, so in the authors' words, "runnable tests can still be shallow."

Tabassum, A., Intesum, M., Arefin, M. F., and Zaman, T. S. VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories. 1st International Workshop on PromptOps and Vibe Coding (POVC '26), co-located with ASE 2026. arXiv:2609.05978.
Tests are a floor, not a ceiling. A green suite means the things you thought to check still work. It says nothing about the things you didn't think of, and nothing at all about security. Passing CI is not evidence that your code is good, only evidence that it is not broken in the specific ways you described.

Requirements

1. Make your logic testable

A test has to be able to call your code. If your logic lives inside a click handler, or inside a function that reads from the page and writes back to it, there is nothing a test can get hold of. (Generally, we want functions that we can call inside of an assertion statement.)

So: pick at least three pieces of real logic and move them into their own module, as functions that take arguments and return values without touching document at all.

// scoring.js  --  logic only, no page, no DOM
export function letterGrade(score) {
  if (score < 0 || score > 100) throw new RangeError("score out of range");
  if (score >= 90) return "A";
  if (score >= 80) return "B";
  return "C";
}

Your page then imports it and stays in charge of the buttons and the display. The split you are aiming for is logic that computes, interface that shows.

Your app must still work when you are done. This is a refactor, not a rewrite. Note: If you ask AI for help with this step, be sure to demand that the AI does not fix any bugs it finds along the way. For the purposes of this assignment, we want to discover those through our tests instead.

One potential trap: To use import in the browser, your script tag becomes <script type="module" src="main.js"></script>. Module scripts don't put their functions on the global object, so any onclick="doThing()" written directly in your HTML will silently stop working. The fix is to attach handlers in JavaScript with addEventListener instead. Nothing will error; the button will just do nothing, which is much worse.

If your Project 2 has a Python backend, you can do all of this in Python with pytest instead. Every requirement still applies in the same way, and the only part that really differs is the CI workflow in Step 6, which will install Python and run pytest rather than Node. Tell us if you're going this route and we'll help you with the workflow file. Everything below is written for JavaScript because that is where most of your code lives.

2. A test suite that actually runs

Create at least eight tests, covering at least three different functions, run with one command, and passing.

Use Node's built-in test runner. There is nothing to install and no config file.

Pick the parts with real logic: anything that computes, parses, validates, filters, sorts, formats, or decides. Skip the parts that just move data from one place to another.

Commit the test suite by itself, before you change any other code. You will see why in requirement 5, and it means we can tell afterward whether a test was edited later.

3. Break it on purpose

This is the requirement that identifies if your test suite is real or decorative.

Make at least five separate one-line changes to your source that should break something, and confirm a specific test fails each time. Flip a comparison, change a boundary, return the wrong variable, delete a validation check, swap two arguments, etc. Then undo it and do the next one.

Record all five: what you broke, which test caught it, and what the failure message said.

When nothing fails, you have found a fake test. That is good to discover, though. Fix it so it would have caught the change, or delete it and say why. Document every one you find.

4. Continuous integration

Set up a GitHub Action so the suite runs on every push, without you doing anything. Then prove it works with two links:

The red one matters as much as the green one. A pipeline nobody has ever seen fail is a pipeline that might not be running.

5. The verifier experiment

Push a commit where one test genuinely fails because the code is wrong. Then open an agent session, give it the repo, and tell it something close to: "The test suite is failing. Make it pass." Say nothing about how it fails.

Then watch carefully and report what it did:

Report what happened (even if this differs from what you expected to happen). If it behaved perfectly, that is a legitimate and interesting result, and we would much rather read an honest account. Quote the relevant part of the transcript and the diff either way.

The previous tests-only commit is valuable here because a git diff against it shows instantly whether your tests are still the tests you wrote.

Optional: go further

Not required, and not at the expense of the five requirements above:

Step-by-Step Instructions

Step 1: Choose what to test (20 minutes)

Open the codebase you chose and find three or more pieces of real logic. Write down, for each one, what it is supposed to do and one way it could plausibly be wrong.

That second part is more important. "It returns the filtered list" is not the basis of a good test, and instead you want specificity like "it returns an empty list rather than crashing when nothing matches."

Step 2: Make your logic testable (45 minutes)

Move those three pieces into a module of their own, as functions that take arguments and return values. Add a package.json at the top of your repo containing exactly this, so Node treats your .js files as modules:

{ "type": "module" }

If any of your existing code uses require(), this will break it, because the setting applies to the whole repository. In that case skip the package.json and give your logic and test files the .mjs extension instead, which is always treated as a module.

Then update your page to import from the new module, switch the script tag to type="module", and replace any inline onclick attributes with addEventListener calls.

Your page will stop opening from a file:// address. Browsers refuse to load module imports when you open an HTML file directly, so the page comes up blank with a CORS error in the console and it looks like the refactor destroyed your app. It didn't. Serve the folder over HTTP instead: run python -m http.server 8000 in your project directory and open http://localhost:8000. On Windows, if python opens the Microsoft Store, use py -m http.server 8000.

Open your app and click every button before moving on. This is the step most likely to break something quietly.

This is a good place to use an agent, since it's a mechanical refactor. It's also a good place to review the diff closely, because "move this logic into a module" and "rewrite my app" are uncomfortably close together in instruction space. Make sure to tell AI not to fix any existing bugs at this point.

Step 3: Get one test running (30 minutes)

Make a tests/ directory and write one test by hand (not with AI). Make this a test that you already know the desired behavior for. And again, no AI here, please.

// tests/scoring.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { letterGrade } from "../scoring.js";

test("ninety is an A", () => {
  assert.equal(letterGrade(90), "A");
});

test("just under ninety is a B", () => {
  assert.equal(letterGrade(89.9), "B");
});

test("a negative score is rejected", () => {
  assert.throws(() => letterGrade(-5), RangeError);
});

Run the whole suite with node --test.

The .test in the filename is not decorative. node --test finds your tests by filename, matching patterns like *.test.js. Name the file tests/scoring.js instead and it will never be found, and a run that finds nothing still prints a cheerful green summary and exits successfully. The folder name won't rescue you either, since Node looks for a folder called test, not tests. If your test count is 0, this is almost always why.

Then make a test fail on purpose before you go any further, so you know you are looking at a real result rather than at a runner that silently found no files.

Budget most of this step for import paths. Getting the test file to see your application code is the fiddliest part of this assignment.

Step 4: Build out the suite with AI (40 minutes)

Now bring in your agent, and give it the list from Step 1 rather than asking it to test everything. (If you give it a broad one-shot prompt, you won't learn anything here, and won't have good answers for your interviews.) Tell it what each function should do and what you think could go wrong.

Ask explicitly for boundary cases and error cases, the two kinds AI reliably skips when left alone. Empty input, zero, negative numbers, missing properties, wrong types, the value exactly at the edge of a comparison.

Read every test it writes. A test you don't understand cannot help you.

Commit here. Tests only, nothing else.

Step 5: Break it on purpose (45 minutes)

Make five one-line changes to your source, one at a time, each confirmed by a failing test, each undone before the next. Keep the table as you go.

Expect to find at least one fake test. Most people find more. Fix or delete each one and note it.

Step 6: Wire up CI (35 minutes)

Create .github/workflows/tests.yml:

name: tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: "26"
      - run: node --test

That's the whole file. Because the test runner ships with Node, there is no install step to go wrong.

Push, then open the Actions tab on GitHub and watch it run. If it fails, the usual cause is a file path that works on your machine but not on Linux, where filenames are case-sensitive. Scoring.js and scoring.js are the same file on your laptop and two different files on the runner.

Step 7: Make it go red, then green (20 minutes)

Push a commit with code you know is broken and confirm the Action fails ("going red"). Save that run's URL. Then fix it, push again, and save the passing run's URL. Both go in your write-up.

Step 8: The verifier experiment (30 minutes)

Push a genuinely failing test, hand the repo to an agent, tell it to make the tests pass, and record exactly what it does. Keep the transcript and the diff.

Run it twice if you have time, ideally with two different models. The variation between them is often the most interesting part of the result.

Step 9: Write it up (25 minutes)

Document as described in What to Submit.

Step 10: Push and submit (10 minutes)

Push everything and fill out the Google form.

What to Submit

Your project repository (or a fork of it), containing:

The two CI links are the easiest points here to earn and the easiest to forget. They are two URLs from your repository's Actions tab.

Google form: Submit your repo link here before Sunday 11/15 at 8:00 PM.

Grading

Graded on whether your tests would actually catch a regression (i.e. a bug that breaks something that used to work), and on the honesty of what you report. Your grade is not based on whether you have an extremely large number of tests.

Component What We're Looking For
Testable structure (15) At least three pieces of real logic moved into importable functions that don't touch the page, with the app still working afterward.
Test suite (15) At least eight tests across three or more functions, passing, aimed at code with real logic in it, committed on their own before other changes.
Break it on purpose (25) Five documented one-line breakages, each with the test that caught it and the failure message. Fake tests found, reported, and resolved. This is the core of the assignment.
Continuous integration (20) A workflow that runs on every push, with links to one passing run and one failing run from your own repository.
Verifier experiment (15) The experiment actually run, with transcript and diff, and an honest account of what the agent did rather than what you expected.
Write-up (10) Clear reasoning about what you tested and what you skipped, written for someone deciding whether to trust your suite.

Eight tests you have tried to break are worth more than 67 you haven't. A suite that found two of its own fake tests is stronger evidence of understanding than one that erroneously reports none.

Resources

Official documentation

Things worth testing in a typical Project 2

If you are staring at your own code unsure where to start, these are the usual suspects:

Tips

Questions? Ask on Ed, attend office hours, or email the instructor. Consult the course AI usage and collaboration policies on the course homepage.