CMU 15-114: Effective Coding with AI
Goal: Build a small app around a vector database. Your app should let a user add new information to the database while the program is running, search that database by meaning rather than by exact words, and then do something genuinely useful with what comes back.
Why This Matters: A traditional database finds rows that match what you typed. A vector database finds things that mean something similar to what you typed, even when none of the words overlap. This type of database gets leveraged by a lot of the AI products you've used recently, and can help with building searches that understands paraphrases, assistants that remember what you told them last week, recommendation systems, duplicate information detection. Once you've built one yourself, a fair amount of apparently magical AI behavior stops being mysterious.
What's different from a normal database app: You don't write SQL or design a schema for this assignment. You'll be storing text by letting an embedding model turn that text into numbers and asking the database for the items closest in meaning to a query. Rather than specifying columns and performing very concrete searches, the questions you'll consider are more like "what should I store, how should I break it up, and what do I do with the top few results?"
You do not need full CRUD on this one. Update and delete are optional. Adding and retrieving are the required halves.
We recommend Python with ChromaDB. It installs with pip, stores everything in a local folder, and it will embed your text for you by default. This is super helpful because it means you don't need an API key to get a working database. You're welcome to use a different vector database if you have a reason to, but we'll be best equipped to help you with Chroma.
A command-line interface is completely sufficient for full credit. As with our other database work, the interesting part is behind the interface. Build a CLI first. If you have time left over and want to put a web front end (or some other sort of graphical UI) on it, go ahead, but get the core working first.
Pick something you would actually use. Here are some directions, but we're leaving them underspecified on purpose:
One question you can optionally ponder: how might a vector database be useful to a chatbot? If a program can store things as it goes and later retrieve the ones most related to what is being discussed, what does that let the program do that it otherwise couldn't? We aren't going to answer that for you. We suspect several of you will land somewhere interesting on your own.
You need a starting corpus (i.e. a body of text data) that's big enough to justify searching. Some places to look, all free:
.gitignore and tell us in your README where to download it and how to load it.
This is the requirement you're most likely to miss! It is entirely possible to build a working app on top of a vector database that did not need one, and that app will not receive full credit.
Before you build, write down two realistic queries a user might give your app where:
For example, in a recipe collection, the query "something warm for a sick day" should surface a chicken soup entry that never uses the words "warm," "sick," or "day."
A "query" here doesn't have to be a typed question. If your app works by matching one whole item against the rest of the collection (a recommender, or a near-duplicate finder), then your query is that item, and the test is the same: it should find its match without sharing the obvious words.
If you can't come up with two such queries, your idea is probably better-suited to use a regular database. Pick a different idea, or change what you store so that meaning actually matters.
Your README must include your two queries, the items they correctly retrieve, and one sentence on why a keyword search would have failed. Demonstrate at least one of them in your video. This is how we check requirement 4, and it's quick to write once your app works.
README.md and located at the repository root (or inside the project folder if you placed the project in your portfolio repo). Include:
prompt_log.txt or prompt_log.md, and located in the same folder as your README. It must list which AI model(s)/tools you used, document the development process from start to finish, and include important, non-trivial prompts verbatim rather than AI-written summaries of them. As a whole, this file should make it obvious that you invested roughly 4.5 hours of work.
Include a sentence or two on which model(s) or tool(s) you used for which parts of the work, and why.
| Grade | (Roughly) What We're Looking For |
|---|---|
| A (90-100) | A real corpus is loaded and the data persists between runs. The user can add new items at runtime and search them immediately. The app does something genuinely useful with what it retrieves. Your two queries convincingly demonstrate that a keyword search wouldn't have worked. Clean, understandable code, thoughtful README and prompt log, evidence of ~4.5 hours of effort, and some creativity in what you chose to build. |
| B (80-89) | All the required pieces work, but the app is a thin wrapper around "search and print the results," or the case that a vector database was the right choice here is weak. Documentation present but less detailed. |
| C (70-79) | A user can add items and retrieve them by similarity, but little is done with the results, or the data doesn't survive a restart, or a plain keyword search over a normal database would have worked just as well. Basic README. |
| Below C | Incomplete or non-functional submission. |
Note: We've seen a few prevalent shortcuts taken in previous assignments that seem to be backfiring on people, either by producing substandard results or circumventing your learning goals. To that end, please avoid the following, and note that we may assess deductions for these:
We recommend working through these steps in order. Each one builds on the last, and this order will help you make steady progress rather than getting stuck debugging several things at once. If you find that one of these steps is taking significantly longer than the time we specified here, or if you aren't confident how to start, please ask on Ed or come to office hours!
Pick an idea, then run the test from the Earning Its Place section on it. Write your two queries down now, in a scratch file, before you write any code. They will shape what you decide to store, and you need them in your README anyway.
Pick your corpus at the same time, since the two decisions constrain each other. Go look at the actual data before you commit to it: open the file, see how long a typical entry is, and decide whether one entry is the right size to be one item or whether you'll need to break it up.
Also decide, in one sentence each: what counts as a single "item" in your database, and what does your app actually do with the items it retrieves?
Make a new folder (and a virtual environment if you wish), then pip install chromadb. Fair warning that this is a somewhat large install (it brings along a machine learning runtime and its dependencies, a few hundred megabytes all told) and it is not fast, so start it and read ahead while it works.
Then write five lines that create a client and a collection, and run them. Don't build anything yet. You just want to confirm the library imports and your environment is sane before you start putting real code on top of it.
The general shape, using Chroma:
import chromadb
# PersistentClient writes to disk. The default in-memory client does NOT.
client = chromadb.PersistentClient(path="./chroma_store")
collection = client.get_or_create_collection(name="notes")
collection.add(
documents=["Chicken soup with ginger and garlic, good when you're run down"],
metadatas=[{"category": "dinner"}],
ids=["1"],
)
print(collection.count())
Add ten or fifteen items by hand. This is a smoke test, not your real data: you want to confirm the whole path works on something small before you point it at a few thousand records. Then quit your program and run it again, and confirm collection.count() still reports your items. If it reports zero, you're using an in-memory client.
Now replace the hand-typed items with your actual corpus. Three things to get right:
collection.add() takes lists. Passing a few hundred documents in one call is dramatically faster than looping and calling it a few hundred times.Attach useful metadata as you go (source, date, category, whatever your data has). You'll want it later for filtering, and adding it now is much easier than backfilling.
Run a few queries and print the whole result object before you do anything clever with it:
results = collection.query(query_texts=["something comforting to eat"], n_results=3)
print(results)
Spend real time on this step. Notice that results come back as lists of lists (one inner list per query), which surprises almost everyone the first time. Notice the distances, where lower means more similar. Then try a query that should match nothing at all, and look at the distance you get back, because Chroma will cheerfully hand you your three closest items even when all three are irrelevant. Knowing roughly where the boundary sits for your data is what lets your app behave sensibly later.
This is also the moment to run the two queries you wrote in Step 1 and find out whether your idea works the way you assumed it would.
Wire up the input path: the user types (or pastes, or points at a file) something new, it goes into the collection, and you can search for it immediately afterward.
Two things bite people here. First, ids have to be unique, and if you reuse one, Chroma ignores the new record without raising an error, so your addition silently never lands. (If you actually want to replace an existing item, the method for that is upsert(), not add().) Second, whatever you store as the document text is the thing that gets embedded, so if you store a title and hide the real content in metadata, similarity search has almost nothing to work with.
This is the heart of the assignment and the step most worth your time. You have the closest few items to whatever the user asked for. Now make them useful. Depending on your idea that might mean composing an answer, ranking and explaining, flagging a near-duplicate, suggesting a next thing, assembling material for something else to work with, or something we haven't thought of.
Resist the urge to stop at printing a list. The difference between a B and an A on this assignment is mostly decided right here.
Build a menu loop around the pieces you already have working, something like:
--- My Notes ---
1. Add something new
2. Ask a question
3. Show everything stored
4. Quit
Choose an option:
Then test the full flow end to end: add several items, search, add another, search again, quit, relaunch, and confirm everything is still there.
Run your two queries from Step 1 against the finished app and capture what comes back. If they don't work as well as you hoped, that is useful information, and you have options: store different text, store more context per item, break items up differently, or change how many results you pull. Write the final versions into your README along with the one-sentence explanation of why a keyword search would have failed.
Document your work as described above (see "What You Must Submit"). Remember that these are two separate files.
Make a short screencast (a minute or two is fine) showing a user adding something, searching, and your app doing its thing with the results, including at least one of your two queries. Push everything to GitHub, add it to your portfolio, and fill out the Google submission form. Budget real time here: uploads and permission settings take longer than people expect, and you still need to test your link in an incognito window.
A quick reference for the ideas you'll need. We'll cover these in lecture, but this is here for you to refer back to.
n_results in Chroma). Note that there is no built-in "no good matches" answer. If you want one, you have to look at the distances and decide on a cutoff yourself.PersistentClient(path=...).chromadb.Client(Settings(...)), or collection.persist(), or passes arguments the library rejects, you've been given code for a version you don't have. Check what you installed with pip show chromadb and compare against the current docs. This is the most likely place for you to lose time on this assignment, and it's an excellent candidate for your "one place AI got it wrong" writeup.where={"category": "dinner"}). Use vectors for the fuzzy part and metadata for the crisp part.count() tells you. This single change fixes most "why are my results so vague?" complaints.chroma_store/ folder (or whatever you named it) gets rebuilt when someone runs your app, and your source data may be far too large for a repo. Keep both out of GitHub, and explain in your README how the database is built on first run and where to get the data.