HW8: Build Your Own Agent Toolkit

CMU 15-114: Effective Coding with AI

Due Date: Mon. 11/9 at 8:00 PM
Time Estimate: ~6 hours
Deliverables: A GitHub repo containing your toolkit, a demonstration write-up, and a README.md, plus a Google form submission
You will need an agentic tool, not a chat window. Claude Code, Codex (included with your ChatGPT Plus account), Cursor, Copilot agent mode, or Gemini CLI all work. If you don't already have this set up, do that ASAP and ask us if you need anything.

Assignment Overview

The premise: At this point you've probably started noticing some instructions that you repeatedly give your AI when doing assignments. "Don't rewrite files I didn't ask you to touch." "Check this against my acceptance criteria." "I need to follow these course policies." Building off of Jack's earlier preview, we're going to make reusable tools that will encode these behaviors for your agents.

You'll build a small, reusable agent toolkit: a skill that configures your AI tools to work the way you have learned you want them to work, and an MCP server that gives them something new they can do. Then you'll prove it does something by using it to do a task and iterating through failures.

Why This Matters: Up to now you've been a user of AI tools. Configuring them is the next step, and it's where most of the leverage is for people who do this professionally. A good toolkit means you stop re-explaining your standards and instead start building a harness that works well for you, specifically you.

This is also preparation for Project 3. Your capstone is completely open-ended and will be the largest thing you build this semester. Build this toolkit so that future-you actually benefits from it.

Learning Objectives:

Skills and MCP Servers

A skill is a folder with a SKILL.md file in it. That's genuinely all it is. The file has a little YAML frontmatter (a name and a description) followed by markdown instructions telling the agent how to do something.

my-skill/
├── SKILL.md          # required: metadata + instructions
├── scripts/          # optional: code the skill can run
├── references/       # optional: documentation it can consult
└── assets/           # optional: templates

Your agent reads only the name and description at startup, so it knows roughly what each skill is for. When a task looks like a match, it loads the full instructions. That's why the description matters so much: it's the part that determines whether your skill ever gets used.

The format is an open standard, originally from Anthropic and now maintained under the Linux Foundation's Agentic AI Foundation. The same folder works in Claude Code, Codex, Cursor, Copilot, Gemini CLI, and a long list of others, which means the toolkit you build this week is not locked to one vendor.

A worked example

Here's a complete, useful skill. It reviews a project against the acceptance criteria in a SPEC.md, which should look familiar from HW7.

---
name: spec-check
description: Reviews the current project against the acceptance criteria in
  SPEC.md and reports which are met, partially met, or not met. Use when
  checking progress against a spec, or before submitting work.
---

# Spec Check

## What to do

1. Read `SPEC.md` in the project root and find the numbered acceptance criteria.
2. For each criterion, locate the code responsible and decide whether it is met.
3. Do not fix anything. Report only.

## Output format

One line per criterion:

`[MET | PARTIAL | NOT MET] #<n>: <criterion> — <evidence, naming file and function>`

Then list the three highest-priority gaps.

## Rules

- Cite a file and a line for every MET claim. If you can't cite one, it isn't met.
- Judge what the code does, not what the comments or README claim it does.

We expect this to work because it has a narrow job and a specified output format. Also, the two rules are meant to avoid common failure cases: an agent that quietly fixes things while reviewing and an agent that believes the README. Good skills are mostly made of constraints you learned the hard way. There's no substitute for experience or critical thought here, so take your time and be deliberate when creating these.

And what's an MCP server?

Where a skill tells your agent how to work, an MCP server gives it something new it can do. You write a small program that exposes one or more tools, your agent is told where to find it, and from then on it can call those tools the same way it calls a file read or a web search.

A minimal one-tool server in Python is genuinely about this long:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("my-notes")

@mcp.tool()
def search_notes(query: str) -> str:
    """Search my course notes and return the most relevant passages.

    The docstring is very important: this is what the agent reads to decide
    whether to call this tool, so describe it the way you'd describe
    it to a collaborator.
    """
    # Replace this with your own lookup, e.g. the vector store from HW6.
    results = f"(no results yet for: {query})"
    return results

if __name__ == "__main__":
    mcp.run()

Then you register it with your client (a few lines of JSON or TOML, depending on the tool) and the tool shows up in your agent's available actions. The MCP docs have quickstarts for both Python and TypeScript.

Note the docstring. Like a skill's description, it's the only thing the agent sees when deciding whether your tool is relevant to the task at hand, so it needs to describe what the tool does succinctly and accurately. Also, as you might imagine... if the tool does not do what it says it does, the AI probably won't be able to fix it, and may not even detect it.

Requirements

1. Find your opportunity with evidence

The skills you build for your AI should come from your past experiences and moments of friction. Memory is fickle, so we've been asking you to share your prompt logs for each assignment, and these provide a valuable way for you to reflect on your process. Go read your own prompt logs from HW3 through HW7. Find a thing (like an instruction or a chunk of guidance) you typed more than once, or a mistake you corrected more than once, and plan on building your skill around that.

Quote the evidence in your write-up. We don't need much; just two or three lines from your old logs is plenty.

2. One skill and one MCP server

You're building two things to solve two kinds of problems. Part of the goal of the assignment is to distinguish which kind of solution you need:

When you want different behavior, write a skill. When you want new reach, build an MCP server. These are easily conflated.

The skill: a checker

Build a skill that reviews work against a standard and reports without fixing anything. The spec-check example above is one kind; a security checker, an accessibility checker, or a "which files did the agent touch that I didn't ask it to" checker are others.

Run your checker in a separate agent session from whatever produced the work, with a fresh context, and ideally on a different model. An agent reviewing its own output is an unreliable reviewer, for roughly the reason you are a bad proofreader of your own writing. Separating the two is the most valuable habit in this assignment.

The MCP server: one working tool

Build a small MCP server exposing one tool that your agent can call. One is enough. Do not build five, and it does not need to be hosted anywhere: a local server over stdio (the Standard Input/Output streams on your computer) is exactly what we're asking for.

Make it something you'd actually use. Some options:

The hard part of MCP is the registration. Writing a one-tool server is usually only about thirty lines. Getting your client to find it, start it, and actually call it is the part where you can get genuinely stuck for a long time, and the config format differs by tool. We'll do this step together in class on Wednesday using the Codex plugin in VS Code since you've all got access. Budget your time accordingly if you miss it, and plan to ask us for help.

3. A demonstration on real work

Use the toolkit (both the skill and the MCP tool) on an actual task: the agent calls your MCP tool, and your checker reviews the result in a separate session. Perform two runs minimum, so you can show it behaves consistently rather than having gotten lucky once. Record what the agent did.

The task can be anything real: a piece of a past assignment redone properly, groundwork for Project 3, something for another class, a personal project.

4. One documented failure and revision

This is the requirement that matters most. Your skill will not work correctly the first time. Find the place where it misfired (it was ignored, it was too vague, it was too rigid, it triggered when you didn't want it to), and fix it. Then document the before, the failure, and the after.

A skill that was never revised is a skill that was never really tested. We would much rather see an honest "version 1 was ignored completely because my description was too vague, here's version 2" than a polished skill with no history.

Optional: go further

Not required, and don't attempt these at the expense of the four requirements above:

Step-by-Step Instructions

Step 1: Find your opportunity (30 minutes)

Open your prompt logs from this semester and search for repetition or friction in your process.

Write down two things: a piece of guidance or instructions you keep having to restate across projects (that becomes your checker skill) and something your agent kept not knowing (that becomes your MCP tool). For each, note the evidence from your logs.

Then spend a few minutes browsing existing skills (see Resources) to see how other people write them, and to check whether someone has already solved your problem well. Borrowing and adapting is fine and encouraged, as long as you say what you borrowed and add something uniquely yours.

We'll do this step together in class on Monday, so if you were there, this is mostly already done.

Step 2: Set up your skills directory (15 minutes)

Find where your tool looks for skills and put a folder there. This varies by tool and changes often enough that you should check your tool's current documentation rather than trusting a blog post. Note that asking AI will also be unreliable here. They don't always find the most recent documentation. The Resources section has the official links.

Write a trivial skill first (one that does something obviously visible, like always starting its reply with a specific word) and confirm your agent actually picks it up. Do this before building anything real. Half the difficulty in this assignment is discovering that your agent was never reading your skill at all.

Step 3: Write your checker skill (45 minutes)

Keep SKILL.md short. A few hundred lines is a lot; most good skills are well under one hundred. Put long reference material in a separate file and point at it.

Spend real effort on the description field. It is the only thing your agent sees when deciding whether your skill is relevant, so write it to say both what the skill does and when to use it.

Specify the output format explicitly, and include at least one rule that prevents a known failure (e.g. "do not modify any files," "cite a file and line for every claim").

Step 4: Build your MCP server (75 minutes)

Start from a quickstart in the MCP docs and get a single tool returning something, anything, before you wire in real data. Run it standalone first and confirm it starts without errors.

Write the tool's docstring with as much care as the skill's description. Remember that this is what the agent reads when deciding whether to call it.

Step 5: Register it and confirm your agent calls it (30 minutes)

Add your server to your client's config, restart the client, and verify the tool appears in the agent's available actions. Then ask a question that should make it call your tool, and confirm it actually did rather than answering from its own knowledge.

The format differs by client, so check your tool's current docs. For Codex, the CLI and the VS Code extension share one file at ~/.codex/config.toml, and a stdio server is three lines:

[mcp_servers.my-notes]
command = "python"
args = ["C:/full/path/to/my_notes_server.py"]

In the extension you can open that file from the gear menu under MCP settings → Open config.toml. Use forward slashes even on Windows, since backslashes need escaping in TOML. Claude Code, Cursor, and the rest use JSON instead, but they all want the same three things: a name, a command, and the arguments to pass it.

This is the step most likely to go wrong. Common causes: a JSON or TOML typo, a relative path where an absolute one was needed, the wrong Python interpreter, or a server that writes to stdout and corrupts the protocol (log to stderr instead).

Step 6: Use the toolkit on real work, twice (60 minutes)

Do the actual task. Work in one session with the MCP tool available, then run your checker in a separate session with a fresh context. Keep the transcripts or copy the relevant parts out as you go. Reconstructing them afterward is painful.

(OH! There's an interesting idea here... while it doesn't quite fit the description of this assignment, it could be very useful to have a skill that aggregates your prompt logs for you as you go. Try this if you want, in addition to making the skill that you've already imagined. We might also do this in class.)

Step 7: Break it, then fix it (45 minutes)

By now you have probably already seen something misbehave, from either piece. If you somehow haven't, go looking: use the skill on a task slightly outside what you designed for, hand it to a different model, or feed your MCP tool an input you didn't anticipate. Then revise, and rerun to confirm the fix.

Step 8: Write it up (45 minutes)

Document as described in What to Submit. Write this for future-you during Project 3, who will have forgotten the details.

Step 9: Push and submit (15 minutes)

Push to GitHub and fill out the Google form.

What to Submit

One new public GitHub repository containing:

These are separate files. README is for someone installing your toolkit, DEMONSTRATION is the evidence, ITERATION is the story of fixing it. Don't merge them.
Note that we are not requiring a prompt log for this assignment. That's because most of what you'll be doing here involves making your own decisions and specifying what you want (rather than what the AI wants). You are not explicitly restricted from using AI to help structure your skills/MCP tools, but a tool with egregiously AI-written text may earn less points if we feel like you haven't been thoughtful about your decisions and wording.

Google form: Submit your repo link here before Sunday 11/8 at 8:00 PM.

Grading

Graded on whether your process and documentation are thoughtful, and whether your toolkit is real, grounded, and tested (not on how clever it is).

Component What We're Looking For
The skill (20) Create a working checker with a narrow job, specific instructions, a defined output format, and a description written so the agent loads it at the right time.
The MCP server (25) Create one tool that works, register it with your client, and demonstrate that it can be called by your agent. Provide installation instructions good enough for us to run it.
Grounding (15) Provide evidence and justification for your toolkit from your own past prompt logs.
Demonstration (20) Use your toolkit on something real, with the checker in a separate session. Run this twice with iteration. Your transcripts must show what the agent did, including where it made the tool call.
Iteration (20) Describe a failure or a place where improvement was necessary, how you found and fixed it, and the results of a rerun. Heavily weighted relative to its size.

A toolkit that is modest but genuinely used and genuinely revised will score better than an elaborate one with no evidence behind it, or one that's thoughtlessly lifted from online or vibe-coded in a hurry.

Speaking of being in a hurry... a few of you won't read the assignment writeup and won't notice that 55 points come from documentation, but so it goes.

Resources and Examples

Official documentation (trust these over blog posts)

Skill ideas at about the right size

These are deliberately mundane. Mundane and used beats impressive and ignored.

BIG SECURITY WARNING: BE CAREFUL WHAT YOU INSTALL. Skills can bundle executable scripts, and your agent may run them. In February 2026, a Snyk study analyzed 3,984 published skills and confirmed 76 of them contained malicious code. That's roughly two percent. Read any third-party skill before you install it, especially anything in scripts/. If you wouldn't run a stranger's shell script, don't install their skill.

Tips

Questions? Ask on Ed, attend office hours, or email the instructor. Consult the course AI usage and collaboration policies on the course homepage.