Additional course information available on Canvas.
Syllabus
Course Goals
The learning goals of the course are as follows:
- To introduce basic concepts of NLP and HCI, to the extent that students can read research papers in the relevant field.
- To introduce a variety of emerging topics related to human-centered NLP.
- To introduce practical tools for building & analyzing, NLP models, and building interactive systems.
Notice that this new course is mostly designed to be a graduate-level, semi-seminar-style course for students interested in HCI+NLP research. This means:
- The course is largely project-oriented, and does not involve exames.
- A large proportion of your grade will depend on research paper reading and digestion.
- This is a special topic course (not well-established) and may only be offered once, so please expect some glitches in the schedule or assignments :)
- I cannot guarantee it will count as a part of a technical requirement.
Prerequisites
There is no explicit prerequisite; However, students are expected to (1) be proficient in Python (for completing assignments), and (2) know basic ML concept — To the extent that you understand concepts like train/dev/test set, model fitting, feature, supervised learning, etc. (We will not cover these in this course!)
If you are familiar with NLP and relevant programming libraries (e.g. HuggingFace), you might find certain parts of the course introducing NLP concepts significantly easier (or, unnecessary :D).
Course Materials and Communications
- Slides and readings will be on this page.
- Assignments and discussions will be on Canvas when their time comes. The course webpage will have links that direct you to them.
- If you have questions related to course materials or logistics, please post on Canvas Discussion page, so either the instructor or other students can help answer that (I will check Discussion regularly).
- If you have special requests, please email the instructor at sherryw@cs.cmu.edu.
- This is an in-person class by default, but if you cannot come, there’s a zoom option on Canvas.
Major Research Work
Grade
The tentative breakdown for grading is below. As a reminder, here is the university policy on academic integrity. See major course work
- 36% Homework Assignments
- 30% two assignments
- 6% two peer grading
- 40% Final Project
- 5% Form group + short project description
- 10% Midterm presentation
- 15% Final presentation
- 10% Project report
- 14% Paper presentation
- 10% Participation and Discussions (Through discussions on Canvas)
Late Day Policy: Attending class in person and submitting course deliverables on time is critical. However, we realize that things happen, and that you might sometimes not be able to turn in your assignments. To accommodate this, you will each receive 4 free late days. Beyond those days, you receive a 5% penalty for each day late. You are welcome to budget late days as you like for the two assignments, two grading deadlines, and the final report.
Project presentation, paper presentation, and paper discussions cannot use late days as they are time sensitive; Final project report cannot use more than two late day as it affects grade submission.
Assignments
-
Model Evaluation (link to opensourced repo). We will evaluate existing Huggingface models with a model testing framework called CheckList. Hopefully the assignment will help you get familiar with the basic programming environment setup for playing with NLP models locally, and the concept of model evaluation.
-
Prompting via Crowdsourcing Strategies. We will apply decades of research findings in Crowdsourcing instructions to LLM prompting. we will select one paper in Crowdsourcing, replicate the idea by writing prompts to instruct different LLM modules as if they are crowdworkers, see whether the crowdsourcing task design strategies can transfer to LLM prompting. Hopefully the assignment will help you get familiar with the concept of LLM prompting, the OpenAI interface, and some crowdsourcing techniques.
Each assignment will have 100 points, and will be peer-graded:
- A base grade up to 80. I will try my best to make this easy to get, as long as you follow the instructions. If you find the assignment difficult, there will also be some ways to earn partial credits.
- A peer-graded point, up to 20. Three students will rate your assignment on some pre-set Likert Scale (e.g. 1 being poorly constructed 5 being very impressive), and your score depends on the relative ranking of averaged your test score usefulness. you will get {20, 15, 10, 5} scores if your averaged score is ranked top {25%, 50%, 75%, 100%} percent respectively.
We will provide more instructions, grading details, and the starter code repo through GitHub Classroom. More details will be posted on Canvas once the assignments are released.
Assignment Peer Grading
You will be assigned to grade other students’ assignments. This is a way for you to see how others think about a given task. You will get full grades for peer gradings unless:
- You do not complete them or submit them late.
- Your score deviates from other people’s rating drastically without justified reasoning (e.g. everyone gave 5 but you gave 1 because you wanted higher ranking).
This is how we will see annotator disagreement in real time :)
Presentations and Discussions
A large proportion of your grade will depend on reading and digesting papers from top HCI or NLP venues. The grading will split into two parts:
-
Paper presentation. The reading lectures will be led by students. 2-3 students will sign up to lead the same session, prepare the slides deck and discuss multiple aspects of the paper. To achieve deep paper digestion, you can take inspirations from the role playing model of Jacobson and Raffel. No need to pick explicit roles, just cover relevant discussion points.
-
Earn participation scores through discussions. Before each reading lecture, we will open corresponding discussion threads on Canvas. Students not leading the session are expected to participate submit comments on those required readings on Canvas. This is how you earn participation scores! Good comments typically exhibit one or more of the following:
- Critiques of arguments made in the papers
- Analysis of implications or future directions for work discussed in lecture or readings
- Clarification of some point or detail presented in the class
- Insightful questions about the readings or answers to other people’s questions
- Links to web resources or examples that pertain to a lecture or reading
Final Project
The most substantial portion of your coursework is a team-based project (2-4 people). You will self-propose a project broadly relevant to HCI+NLP, with four milestones (they will be posted on Canvas when the time comes):
-
Form research group + topic selection. You will fill in a short Google Form that documents your group members, and a general description of your project. This will act as a forcing function for you to start think about the project. In the form, you will mostly address these questions: 1 (what are you trying to do), 2 (how is it done today), 3 (what’s new), 4 (who cares), 5 (your proposed method), and 6 (metrics of success). If you are looking for project partners, please post to Canvas!
-
Midterm presentation + peer feedback. Shortly after the Spring Break, Each group will do a 7-8 minute in-class presentation on the project progress, so the instructor and other students can provide feedback.
-
Final presentation. Each group will do a 7-8 minute in-class presentation on the final project result. This will be similar to the midterm presentation.
-
Final report. Each group will also submit a 4-8 page final report (not counting references) written in the form of a conference paper submission. The paper might include content that is typical of papers that appear at ACL or CHI.
Special Thanks!
- Thanks to Diyi Yang who also created a Human-Centered NLP course at Stanford during a similar time. She taught me the importance of interchanging lectures with reading assignments, helped me revise the course syllabus, and pointed me to awesome resources for various topics.
- Thanks to Haiyi Zhu for inspiring me on the course assignment design!
- Thanks to Jay Alammar and Huggingface tutorial whose interactive visualization on Transformers, interpretability, etc. helped ground the modeling part of the course.
- Lectures on many topics were built upon amazing conference tutorials. Thanks to the organizers for their thorough reflection on specific topics, and for allowing me to borrow the materials. Most noteably: EMNLP 2020: Interpreting Predictions of NLP Models; EMNLP 2021: Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection; NAACL 2022: Human-centered Evaluations of Explanations.
- Thanks to the guest speakers for giving expert-covers on several topics. At one point, I’m gonna do a all-guest-speaker course – I can very easily think of someone more knowledgeable than me on every topic!
- I also borrowed specific materials here and there from various other courses and online blog posts. I tried my best to credit everyone in slides. Thank you!
Respect for Diversity
It is our intent that students from all diverse backgrounds and perspectives be well served by this course, that students’ learning needs be addressed both in and out of class, and that the diversity that students bring to this class be viewed as a resource, strength and benefit. It is our intent to present materials and activities that are respectful of diversity: gender, sexuality, disability, age, socioeconomic status, ethnicity, race, and culture. Your suggestions are encouraged and appreciated. Please let us know ways to improve the effectiveness of the course for you personally or for other students or student groups. In addition, if any of our class meetings conflict with your religious events, please let us know so that we can make arrangements for you.
Accommodations for Students with Disabilities
If you have a disability and are registered with the Office of Disability Resources, we encourage you to use their online system to notify us of your accommodations and discuss your needs with us as early in the semester as possible. We will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, we encourage you to contact them at access@andrew.cmu.edu.
Health and Well-being
If you are experiencing COVID-like symptoms or have a recent COVID exposure, do not attend class if we are meeting in-person. Please email the instructors for accomodations.
If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support. Counseling and Psychological Services (CaPS) is here to help; call 412-268-2922 and visit their website at www.cmu.edu/counseling/. Consider reaching out to a friend, faculty or family member you trust for help getting connected to the support that can help. If you or someone you know is feeling suicidal or in danger of self-harm, call someone immediately, day or night:
If the situation is life threatening, call the police. On campus call CMU Police: 412-268-2323. Off campus: 911.
If you have questions about this, please let the instructors know. Thank you, and have a great semester.