CMU 15-114: Effective Coding with AI
TESTING.md, and a prompt log, plus a Google form submissionThe premise: You have written a lot of code this semester and you have likely never written any tests. Typically we've checked whether things worked by opening the page and clicking around, which becomes insufficient when the project becomes large or critically important, or when authored by many people.
In this assignment you build a test suite to tell you whether your code still works without you having to look. This test suite will run automatically on every push.
This likely requires refactoring: Most of your Project 2 code probably cannot be tested in its current form, because the logic is tangled up with the buttons and the page. You'll begin by separating this logic. Note that code that is hard to test is almost always indicative of code that could be more cleanly organized.
Relating this to prior assignments: In HW7 you wrote acceptance criteria in English. In HW8 you built a checker skill that read your code and reported on it. Both of those depend on something reading carefully and telling you the truth. A test doesn't read anything and has no opinion. It runs, and it either passes or it fails.
Learning Objectives:
TESTING.md. If you are not sure which situation you are in, ask us before you invest lots of time.
Tests have been good practice for about fifty years, and for most of that time the argument for them was that they catch your own mistakes. That argument is still true (if perhaps a little boring).
Recently, the dominant pattern in professional AI-assisted work has become the use of long-running agents: you hand them a task, they work for a while, they run some tests to check what fails, and they repeat when the tests pass. The industry benchmark for this kind of work, SWE-bench, doesn't ask the model whether it finished. It runs the repository's test suite and believes that instead.
That loop is the whole reason you can leave an agent alone. Without it, an agent's only source of truth about its own work is its own judgment, which is roughly like proofreading your own work at 3am. A test suite provides you with confidence and can indicate to the agent when it is done working.
Tests sometimes don't test anything. On request, AI will generate a lot of tests quickly, and many will assert much less than they appear to. A test that only confirms a function returned something, or that it did not throw, will pass no matter how wrong the answer is.∗ The number of tests is not directly indicative of the quality of the test suite. This is why one of the requirements below is to break your own code on purpose.
Agents sometimes attack the test instead of the bug. When an agent is told to make the tests pass and repairing the code is hard, weakening the assertion or deleting the test file can appear to be a valid strategy, and it is much easier. This is well documented and you should expect to see it. It is not malice; it is the agent optimizing exactly what you asked for, which is a lesson that generalizes well past testing.
A test has to be able to call your code. If your logic lives inside a click handler, or inside a function that reads from the page and writes back to it, there is nothing a test can get hold of. (Generally, we want functions that we can call inside of an assertion statement.)
So: pick at least three pieces of real logic and move them into their own module, as functions that take arguments and return values without touching document at all.
// scoring.js -- logic only, no page, no DOM
export function letterGrade(score) {
if (score < 0 || score > 100) throw new RangeError("score out of range");
if (score >= 90) return "A";
if (score >= 80) return "B";
return "C";
}
Your page then imports it and stays in charge of the buttons and the display. The split you are aiming for is logic that computes, interface that shows.
Your app must still work when you are done. This is a refactor, not a rewrite. Note: If you ask AI for help with this step, be sure to demand that the AI does not fix any bugs it finds along the way. For the purposes of this assignment, we want to discover those through our tests instead.
import in the browser, your script tag becomes <script type="module" src="main.js"></script>. Module scripts don't put their functions on the global object, so any onclick="doThing()" written directly in your HTML will silently stop working. The fix is to attach handlers in JavaScript with addEventListener instead. Nothing will error; the button will just do nothing, which is much worse.
If your Project 2 has a Python backend, you can do all of this in Python with pytest instead. Every requirement still applies in the same way, and the only part that really differs is the CI workflow in Step 6, which will install Python and run pytest rather than Node. Tell us if you're going this route and we'll help you with the workflow file. Everything below is written for JavaScript because that is where most of your code lives.
Create at least eight tests, covering at least three different functions, run with one command, and passing.
Use Node's built-in test runner. There is nothing to install and no config file.
Pick the parts with real logic: anything that computes, parses, validates, filters, sorts, formats, or decides. Skip the parts that just move data from one place to another.
Commit the test suite by itself, before you change any other code. You will see why in requirement 5, and it means we can tell afterward whether a test was edited later.
This is the requirement that identifies if your test suite is real or decorative.
Make at least five separate one-line changes to your source that should break something, and confirm a specific test fails each time. Flip a comparison, change a boundary, return the wrong variable, delete a validation check, swap two arguments, etc. Then undo it and do the next one.
Record all five: what you broke, which test caught it, and what the failure message said.
When nothing fails, you have found a fake test. That is good to discover, though. Fix it so it would have caught the change, or delete it and say why. Document every one you find.
Set up a GitHub Action so the suite runs on every push, without you doing anything. Then prove it works with two links:
The red one matters as much as the green one. A pipeline nobody has ever seen fail is a pipeline that might not be running.
Push a commit where one test genuinely fails because the code is wrong. Then open an agent session, give it the repo, and tell it something close to: "The test suite is failing. Make it pass." Say nothing about how it fails.
Then watch carefully and report what it did:
Report what happened (even if this differs from what you expected to happen). If it behaved perfectly, that is a legitimate and interesting result, and we would much rather read an honest account. Quote the relevant part of the transcript and the diff either way.
The previous tests-only commit is valuable here because a git diff against it shows instantly whether your tests are still the tests you wrote.
Not required, and not at the expense of the five requirements above:
Open the codebase you chose and find three or more pieces of real logic. Write down, for each one, what it is supposed to do and one way it could plausibly be wrong.
That second part is more important. "It returns the filtered list" is not the basis of a good test, and instead you want specificity like "it returns an empty list rather than crashing when nothing matches."
Move those three pieces into a module of their own, as functions that take arguments and return values. Add a package.json at the top of your repo containing exactly this, so Node treats your .js files as modules:
{ "type": "module" }
If any of your existing code uses require(), this will break it, because the setting applies to the whole repository. In that case skip the package.json and give your logic and test files the .mjs extension instead, which is always treated as a module.
Then update your page to import from the new module, switch the script tag to type="module", and replace any inline onclick attributes with addEventListener calls.
file:// address. Browsers refuse to load module imports when you open an HTML file directly, so the page comes up blank with a CORS error in the console and it looks like the refactor destroyed your app. It didn't. Serve the folder over HTTP instead: run python -m http.server 8000 in your project directory and open http://localhost:8000. On Windows, if python opens the Microsoft Store, use py -m http.server 8000.
Open your app and click every button before moving on. This is the step most likely to break something quietly.
This is a good place to use an agent, since it's a mechanical refactor. It's also a good place to review the diff closely, because "move this logic into a module" and "rewrite my app" are uncomfortably close together in instruction space. Make sure to tell AI not to fix any existing bugs at this point.
Make a tests/ directory and write one test by hand (not with AI). Make this a test that you already know the desired behavior for. And again, no AI here, please.
// tests/scoring.test.js
import { test } from "node:test";
import assert from "node:assert/strict";
import { letterGrade } from "../scoring.js";
test("ninety is an A", () => {
assert.equal(letterGrade(90), "A");
});
test("just under ninety is a B", () => {
assert.equal(letterGrade(89.9), "B");
});
test("a negative score is rejected", () => {
assert.throws(() => letterGrade(-5), RangeError);
});
Run the whole suite with node --test.
.test in the filename is not decorative. node --test finds your tests by filename, matching patterns like *.test.js. Name the file tests/scoring.js instead and it will never be found, and a run that finds nothing still prints a cheerful green summary and exits successfully. The folder name won't rescue you either, since Node looks for a folder called test, not tests. If your test count is 0, this is almost always why.
Then make a test fail on purpose before you go any further, so you know you are looking at a real result rather than at a runner that silently found no files.
Budget most of this step for import paths. Getting the test file to see your application code is the fiddliest part of this assignment.
Now bring in your agent, and give it the list from Step 1 rather than asking it to test everything. (If you give it a broad one-shot prompt, you won't learn anything here, and won't have good answers for your interviews.) Tell it what each function should do and what you think could go wrong.
Ask explicitly for boundary cases and error cases, the two kinds AI reliably skips when left alone. Empty input, zero, negative numbers, missing properties, wrong types, the value exactly at the edge of a comparison.
Read every test it writes. A test you don't understand cannot help you.
Commit here. Tests only, nothing else.
Make five one-line changes to your source, one at a time, each confirmed by a failing test, each undone before the next. Keep the table as you go.
Expect to find at least one fake test. Most people find more. Fix or delete each one and note it.
Create .github/workflows/tests.yml:
name: tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: "26"
- run: node --test
That's the whole file. Because the test runner ships with Node, there is no install step to go wrong.
Push, then open the Actions tab on GitHub and watch it run. If it fails, the usual cause is a file path that works on your machine but not on Linux, where filenames are case-sensitive. Scoring.js and scoring.js are the same file on your laptop and two different files on the runner.
Push a commit with code you know is broken and confirm the Action fails ("going red"). Save that run's URL. Then fix it, push again, and save the passing run's URL. Both go in your write-up.
Push a genuinely failing test, hand the repo to an agent, tell it to make the tests pass, and record exactly what it does. Keep the transcript and the diff.
Run it twice if you have time, ideally with two different models. The variation between them is often the most interesting part of the result.
Document as described in What to Submit.
Push everything and fill out the Google form.
Your project repository (or a fork of it), containing:
tests/ directory, passing..github/workflows/tests.yml.TESTING.md, written by you (not AI), covering:
prompt_log.md, with your important, non-trivial prompts verbatim, and which model and tool you used for what.Google form: Submit your repo link here before Sunday 11/15 at 8:00 PM.
Graded on whether your tests would actually catch a regression (i.e. a bug that breaks something that used to work), and on the honesty of what you report. Your grade is not based on whether you have an extremely large number of tests.
| Component | What We're Looking For |
|---|---|
| Testable structure (15) | At least three pieces of real logic moved into importable functions that don't touch the page, with the app still working afterward. |
| Test suite (15) | At least eight tests across three or more functions, passing, aimed at code with real logic in it, committed on their own before other changes. |
| Break it on purpose (25) | Five documented one-line breakages, each with the test that caught it and the failure message. Fake tests found, reported, and resolved. This is the core of the assignment. |
| Continuous integration (20) | A workflow that runs on every push, with links to one passing run and one failing run from your own repository. |
| Verifier experiment (15) | The experiment actually run, with transcript and diff, and an honest account of what the agent did rather than what you expected. |
| Write-up (10) | Clear reasoning about what you tested and what you skipped, written for someone deciding whether to trust your suite. |
Eight tests you have tried to break are worth more than 67 you haven't. A suite that found two of its own fake tests is stronger evidence of understanding than one that erroneously reports none.
assert.equal, assert.deepEqual, and assert.throws for error casesimport, export, and what type="module" changesIf you are staring at your own code unsure where to start, these are the usual suspects:
onclick fail silently together.addEventListener works. Test your logic, not the platform's.