Schedule and Readings

Introduction to Course; What are "Software Structures for User Interfaces"; and Why are user interfaces hard to design and implement?

Please fill out this questionnaire before the first class: https://forms.gle/xLUN2jGZDeYVEGKd6

Slides for Lecture 1 and pdf
Video for Lecture 1 on Canvas (only available to people in the class)

Required Readings:

• Brad A. Myers. "Challenges of HCI Design and Implementation," ACM Interactions. vol. 1, no. 1. January, 1994. pp. 73-83. ACM DL
  Discusses why user interfaces are important, and why they are hard to design and implement.

Recommended Reading:

• Brad A. Myers. "A Brief History of Human Computer Interaction Technology." ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. ACM DL

Start on Homework 1 - html, CSS, and JavaScript

Review of HTML and CSS

Slides for Lecture 2 and pdf

Required Readings:

• There are many great online resources to learn HTML/CSS/JavaScript. Here are a few, but let me know what you find useful!
  • reference doc and tutorials:
    • Mozilla Developer Network: for html and CSS and JavaScript
    • W3 Schools: https://www.w3schools.com
  • Video tutorials: CMU has a subscription to LinkedIn learning, e.g., for html/css and for JavaScript
  • Bigger examples: FreeCodeCamp.org - recommended by a Masters of Computational Data Science student

Development setup, Git and GitHub, HTML/CSS

Slides for all labs will be in this folder on Canvas

Review of JavaScript

Required Readings: See list with Lecture 2

Slides for Lecture 3 and pdf

Input 1: Conventional Input Models for Handling Input Events

Slides for Lecture 4 and pdf

Required Readings:

• The JavaScript resources listed above will cover event handling for Input Events, DOM Events, etc., but here are some extra resources:
  • Mouse Events
  • Touch Events

JavaScript, DOM, and events

Course add Deadline is September 12, 2022.

What is "User Interface Software", Overview of UI Software and Tools

Slides for Lecture 5 and pdf

Optional Readings:

• Brad A. Myers. "Graphical User Interface Programming," chapter 48 of Computer Science Handbook -- Second Edition. Allen B. Tucker, editor in chief. Boca Raton, FL: Chapman & Hall/CRC Press, Inc., 2004. pp. 48-1 - 48-29. pdf  (This is a revision of Brad A. Myers. "User Interface Software Tools," ACM Transactions on Computer-Human Interaction. vol. 2, no. 1, March, 1995. pp. 64-103. ACM DL Reference)
• Brad A. Myers, Scott Hudson and Randy Pausch. "Past Present and Future of User Interface Software Tools," ACM Transactions on Computer Human Interaction, (Mar. 2000). Volume 7, Issue 1. pp. 3-28. ACM DL Reference.

Turn in Homework 1.
Start on Homework 2 - Input Handling

Output 1: Basic 2D Computer Graphics

Slides for Lectures 6 & 7 and pdf

Required Readings:

• The following are available on Canvas as a combined pdf under Special Course Readings:
  • New Foley & van Dam, pp. 25-40, 52-60
  • old Foley & van Dam, pp. 132-134 (color maps)

Advanced Input Event Handling

Output 1: Basic 2D Computer Graphics, continued

Input 2: Declarative input models; "Interactor" (Behavior) Objects in Garnet and Amulet
and
Output 2: Graphics Models: Structured Graphics; Multiple Hierarchies; Object-Oriented Techniques

Slides for Lecture 8 and pdf

Recommended Readings:

• Brad A. Myers. "A New Model for Handling Input," ACM Transactions on Information Systems, vol. 8, No. 3. July, 1990. pp. 289-320. ACM DL Reference.
• Brad A. Myers, Richard G. McDaniel, Robert C. Miller, Alan Ferrency,  Andrew Faulring, Bruce D. Kyle, Andrew Mickish, Alex Klimovitski, and Patrick Doane. "The Amulet Environment: New Models for Effective User Interface Software Development", IEEE Transactions on Software Engineering, Vol. 23, no. 6. June, 1997. pp. 347-365. local pdf or IEEE DL.

Optional Readings:

• David S. Kosbie, Brad Vander Zanden, Brad A. Myers, Dario Giuse. "Automatic Graphical Output Management", in Brad A. Myers, editor. The Garnet Compendium: Collected Papers, 1989-1990. Carnegie Mellon University School of Computer Science Technical Report, no. CMU-CS-90-154, August, 1990. pp. 30-43. pdf.

Graphics 1

UI Software Models: Lexical-Syntax-Semantics, Seeheim Model, MVC, Design Patterns, State Diagrams

Slides for Lecture 9 and pdf

Recommended Readings:

• Glenn Krasner and Stephen T. Pope, "A Cookbook for Using the Model-View-Controller User Interface Paradigm in Smalltalk-80", Journal of Object-Oriented Programming (JOOP). August-September, 1988. vol. 1, no. 3. pp. 26-49. pdf scan at UCI

Optional Readings:

• Mark Green, "A Survey of Three Dialogue Models," ACM Transactions on Graphics (5)3, Jul, 1986, pp 244-275. ACM DL Reference
• William Buxton, "Lexical and Pragmatic Considerations of Input Structures," Computer Graphics, January, 1983, (17)1, pp. 31-37. (lexical, syntactic, semantic, etc.). html
• chapter 10 from Olsen's book: "Editing Dialog Models". User Interface Management Systems: Models and Algorithms.Morgan Kaufmann; (November 1991) (pdf of the chapter (CMU only), also avaialble in Special Course Readings on Canvas), or Amazon listing for book.

Turn in Homework 2.
Start on Homework 3 - Graphics using SVG and Canvas

UI Implementation Concepts: Callbacks, Resources, Widget Hierarchies, Geometry Management

Slides for Lecture 10 and pdf

Optional Readings:

• Linton, M.A., Vlissides, J.M., and Calder, P.R., “Composing user interfaces with InterViews.” IEEE Computer, Feb, 1989. 22(2): pp. 8-22. pdf at Georgia Tech
• Funny flash animation about using Java Grid Bag layout (2:42): http://madbean.com/blog/2004/17/totallygridbag.html or YouTube version.

Graphics 2

React for JavaScript, 1

Slides for Lectures 11 & 12 and pdf

Required Readings:

• As with HTML/CSS/JavaScript, there are many great online resources to learn React. The official one is the following (and others are listed below as "Optional"), but let me know what you find useful!
  • ReactJS.org's own tutorial: https://reactjs.org/tutorial/tutorial.html
• To get started with your React app, we recommend using the official way, with create-react-app. Here is some documentation:
  • https://create-react-app.dev/docs/getting-started/ 
• We will be using the package "Router" for React. Here is the official documentation:
  • https://reactrouter.com/web/guides/quick-start

Optional Readings:

• W3 Schools about React: https://www.w3schools.com/react/default.asp
• Mozilla Developer Network: for React
• CMU has a subscription to LinkedIn learning, e.g., for React

React for JavaScript, 2

Same slides as lecture 11, but reload for new content!

React 1

Evaluation of APIs and UI Tools, API Usability, Cognitive Dimensions

Slides for Lecture 13 and pdf

Recommended Readings:

• Brad A. Myers, A. J. Ko, Thomas D. LaToza, and YoungSeok Yoon. "Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools," IEEE Computer, Special issue on UI Design, 49, issue 7, July, 2016, pp. 44-52. IEEE DL or local pdf.
• Brad A. Myers and Jeffrey Stylos, "Improving API Usability", Communications of the ACM, vol 59, No. 6, June, 2016, pp. 62-69, ACM DL; html or local pdf.
• Jakob Nielsen, Heuristic Evaluation. On line in html format. Includes list of 10 heuristics.
• Green, T.R.G. and Petre, M., “Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework.” Journal of Visual Languages and Computing, 1996. 7(2): pp. 131-174. ScienceDirect.
  Includes an analysis of HyperCard

Optional Readings:

• Jeffrey Stylos and Brad Myers, "Mapping the Space of API Design Decisions," 2007 IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC'07. Sept 23-27, 2007, Coeur d'Alene, Idaho. pp. 50-57. ieee or local pdf
• Brian Ellis, Jeffrey Stylos, and Brad Myers. "The Factory Pattern in API Design: A Usability Evaluation". International Conference on Software Engineering (ICSE'2007). May 20-26, 2007. Minneapolis, MN. pp. 302-312. ACM DL or local pdf
• T.R.G. Green, Cognitive Dimensions of Notations. Cognitive Dimensions of Notations Resource Site

Turn in Homework 3.
Start on Homework 4 - Using React

Command Objects & Support for Undo

Slides for Lecture 14 and pdf version

Recommended Readings:

• Brad A. Myers and David Kosbie. "Reusable Hierarchical Command Objects," Proceedings CHI'96: Human Factors in Computing Systems. Vancouver, BC, Canada. April 14-18, 1996. ACM DL Reference
  

Optional Readings:

• Command Objects in Amulet (note: only the overall design is relevant, not the details)
• Brad A. Myers. "Scripting Graphical Applications by Demonstration," Proceedings CHI'98: Human Factors in Computing Systems. Los Angeles, CA, April 18-23, 1998. pp. 534-541. ACM DL Reference and local pdf and video. (Topaz)

Take-home Midterm Exam released Wednesday and due Friday before class

React 2

CMU Mid-Semester Break - no classes

CMU Mid-Semester Break - no classes

CMU Mid-Semester Break - no labs

Mid Semester Grades due: October 24, 2022

Connecting to the backend - Web Services and Databases

Slides for Lecture 15 & 16 and pdf version

Required Readings:

• Documentation for Unsplash, starting here: https://unsplash.com/documentation
• Documentation for Firebase, starting here: https://firebase.google.com/docs

Please fill out the midterm survey: https://www.surveymonkey.com/r/ssui2022midterm
Turn in Homework 4 (deadline extended until Friday, 10/28/22).
Start on Homework 5 - Implement Undo using Command Objects

Connecting to the backend - Web Services and Databases, continued

Same slides as Lecture 15 and pdf

Tartan Community Day; No Classes

Brad away at Dagstuhl

Some Other Web Toolkits - jQuery, AngularJS, Vue, Bootstrap, Svelte, etc.
presented by Alex Cabrera

Slides for Lecture 17 (pdf)

Recommended Readings:

• “Rethinking Reactivity” (video) - https://www.youtube.com/watch?v=AdNJ3fydeao
• Maximilian Schwarzmüller, "Angular vs React vs Vue [2020 Update]", updated March 19, 2020. https://academind.com/learn/angular/angular-vs-react-vs-vue-my-thoughts/

Brad away at Dagstuhl

Designing Large Web Applications
presented by Alex Cabrera

Slides for Lecture 18 (pdf)

Recommended Readings:

• Next.js: https://nextjs.org/learn/foundations/about-nextjs/what-is-nextjs
• Sveltekit: https://kit.svelte.dev/docs/introduction

Turn in Homework 5 (deadline extended until 11/8)
Start on Homework 6 - Connecting to a Backend

Using web services & Remote Databases

Course Drop Deadline is November 7, 2022.

Toolkits for building speech/conversational/chatbot User Interfaces, and Visualizations

• Speech Examples: DialogFlow, Microsoft Bot Network, Voiceflow, ...
• Viz Examples: Prefuse, Flare, Improvise, VTK, Excel, Many Eyes, Google Charts, Tableau, VizQL, ggplot2, Protovis, D3

Slides for Lecture 20 and pdf version

Recommended Readings:

• Google: DialogFlow Quickstart Video https://www.youtube.com/watch?v=9aHusGxntPw&list=TLPQMTMwNDIwMjAF_Z9_ZwIbEA&index=1
• Michael Bostock, Vadim Ogievetsky and Jeffrey Heer. (2011) "D3: Data-Driven Documents", http://vis.stanford.edu/files/2011-D3-InfoVis.pdf. (Paper introducing D3)

Optional Readings:

• Dan Jurafsky and James Martin. Dialog Systems and Chatbots. Chapter of Speech and Language Processing (3rd ed.) https://web.stanford.edu/~jurafsky/slp3/24.pdf
• Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer, "Vega-Lite: A Grammar of Interactive Graphics", IEEE Transactions on Visualization and Computer Graphics, Volume: 23 , Issue: 1 , Jan. 2017, IEEE DL or local pdf
• Bret Victor's "Up and Down the Ladder of Abstraction" (2011): http://worrydream.com/#!2/LadderOfAbstraction (this one is interactive, and really fun)
• Michael Bostock, "What Makes Software Good?" (2016): https://medium.com/@mbostock/what-makes-software-good-943557f8a488. (This has some 'big picture' thoughts on software, but is also a writeup explaining major changes he made to D3)

Toolkit support for Gestural Input Techniques

Slides for Lecture 20 and pdf version

Required Readings:

• J.O. Wobbrock, AD Wilson, Y Li. "Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes", In UIST '07, p. 159-168. ACM DL Ref

Optional Readings:

• Stephen Oney, Rebecca Krosnick, Joel Brandt, Brad Myers. "Implementing Multi-Touch Gestures with Touch Groups and Cross Events," Proceedings SIGCHI '2019: Human Factors in Computing Systems. Glaskow, UK. May 4-9, 2019. ACM DL or local pdf. (Best Paper Honorable Mention Award.)
• The Gesture Recognition Toolkit, http://www.jmlr.org/papers/volume15/gillian14a/gillian14a.pdf
• Hand Gesture Using Kinect, http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6269439

Data Visualization 1

Software Engineering for UIs: How to create high-quality UIs in the context of Agile and other software development processes

Slides for Lecture 21 and pdf version

Recommended Readings:

• Budwig, M., Jeong, S. and Kelkar, K. When user experience met agile: a case study. In Proceedings of CHI'2009: The 27th international conference extended abstracts on Human factors in computing systems (Boston, MA, USA, 2009). ACM, 3075-3084. ACM DL PDF

Turn in Homework 6 (deadline extended until 11/17)
Start on Final Project!

Simple User Interface Toolkits and End-User Programming for UIs; Low-Code / NoCode

• Examples: Scratch, Alice, AirTable, Bubble.io, Zapier, LabView, IFTTT, (older: Yahoo Pipes)

Slides for Lecture 22 and pdf version

Recommended Readings:

• Conway, M., Audia, S., Burnette, T., Cosgrove, D., Christiansen, K., Deline, R., et al. (2000, Apr 1-6). Alice: Lessons Learned from Building a 3D System For Novices. Paper presented at the Proceedings of CHI 2000, The Hague, The Netherlands. ACM DL Ref, or citeseer ref
• See also links in the lecture slides for many more papers and videos about these systems!

Optional Readings:

• Pausch, R., Conway, M., & DeLine, R. (1992). Lesson Learned from SUIT, the Simple User Interface Toolkit. ACM Transactions on Information Systems, 10(4), 320-344. ACM DL Ref, or citeseer ref
• J.F. Pane, B.A. Myers, and L.B. Miller, "Using HCI Techniques to Design a More Usable Programming System," 2002 IEEE Symposia on Human Centric Computing Languages and Environments (HCC'2002). Arlington, VA, September 3-6, 2002. pp. 198-206. local pdf. (Winner Most Influential Paper Award for important influences on VL/HCC research or commerce over the last 10+/-1 years in 2012.)
• Brad A. Myers, John F. Pane and Andy Ko, "Natural Programming Languages and Environments". Communications of the ACM. 47(9), pp. 47-52. ACM DL Ref

Turn in Homework 6 (extended deadline)

Data Visualization 2

Toolkit support for Assistive and Accessible Interfaces; Web Accessibility

Slides for Lecture 23 and pdf version

Recommended Readings:

• W3C, "Introduction to Web Accessibility," on-line
• Zimmermann, G., Vanderheiden, G. Accessible design and testing in the application development process: considerations for an integrated approach. Univ Access Inf Soc 7, 117–128 (2008). https://doi.org/10.1007/s10209-007-0108-6

Optional Readings:

• Gemou, M., Montalva Colomer, J.B., Cabrera-Umpierrez, M.F. et al. Validation of Toolkits for developing third-generation Android accessible mobile applications. Univ Access Inf Soc, 15, 101–127 (2016). https://doi.org/10.1007/s10209-014-0377-9
• Google: Accessibility Android Developers, http://developer.android.com/design/patterns/accessibility.html#tools

Form final project groups

NO CLASS - THANKSGIVING

NO LAB - THANKSGIVING

Toolkits for 3D Programming and the UIs of Games

Slides for Lecture 24 and pdf version

Optional Readings:

• Richard Fine, "Going deep with IMGUI and Editor Customization", December 22, 2015, blog post -- discusses the event loop for Unity's old immediate-mode GUI system
• J. Blow, “Game development: Harder Than You Think”, ACM Queue, ACM, New York, 2004, pp. 29-37. ACM DL REF
• Take a look at standard game engines of today: Unity Game Engine, Unreal Engine, and (for non-programmers) GameMaker. There are dozens of other game engines available.
• OpenGL is the graphics library standard, and it is open source. However, it is highly complex. For a gentler introduction and some videos demonstrating its capability, look at a javascript-based wrapper for OpenGL called three.js.

One-page proposal for Projects due on Wednesday, 11/30/22 at 3:05pm on Canvas

Interactive Tools: Prototypers (HyperCard, Director, Visual Basic, Balsamiq), Interface Builders, Sketching Tools

Slides for Lecture 25 and pdf version

Recommended Readings:

• James Landay and Brad A. Myers. "Interactive Sketching for the Early Stages of User Interface Design," Proceedings CHI'95: Human Factors in Computing Systems. Denver, CO. May, 1995. pp. 43-50. (html) or ACM DL Ref

Optional Readings:

• Brad A. Myers. "Separating Application Code from Toolkits:  Eliminating the Spaghetti of Call-Backs,"  ACM Symposium on User Interface Software and Technology: UIST'91, Hilton Head, SC, Nov. 11-13, 1991. pp. 211-220. ACM DL Reference
• Walker, M., Takayama, L., and Landay, J. (2002) High-fidelity or low-fidelity, paper or computer medium? Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting (in press). 
  PDF
• James A. Landay and Brad A. Myers. Just Draw It! Programming by Sketching Storyboards.   Carnegie Mellon University School of Computer Science Technical Report, no. CMU-CS-95-199 and Human Computer Interaction Institute Technical Report CMU-HCII-95-106. November, 1995. html or pdf

3D programming

Three.JS and basic 3D rendering in a browser

Constraints and Data Bindings

Slides for Lecture 26 and pdf version

Recommended Readings:

• Scott E. Hudson. "Incremental attribute evaluation: a flexible algorithm for lazy update," ACM Transactions on Programming Languages and Systems, Volume 13 , Issue 3 (1991), Pages 315-341.  ACM DL Reference

Optional Readings:

• Brad Vander Zanden, Brad A. Myers, Dario Giuse and Pedro Szekely. "Integrating Pointer Variables into One-Way Constraint Models," ACM Transactions on Computer-Human Interaction. vol. 1, no. 2, June, 1994. pp. 161-213. ACM DL Reference
• Brad Vander Zanden, "An Incremental Algorithm for Satisfying Hierarchies of Multi-way, Dataflow Constraints", ACM Transactions on Programming Languages and Systems, 18(1), January, 1996. pp. 30-72. ACM DL Reference
• Bjorn N. Freeman-Benson, John Maloney and Alan Borning. "An incremental constraint solver," Communications of the ACM . Volume 33 , Issue 1 (1990). Pages 54-63. ACM DL Reference

Please fill out the class survey: https://www.surveymonkey.com/r/SSUI2022Fall-Final and the official CMU survey and the peer survey about your project (required): https://www.surveymonkey.com/r/SSUI2022peer-survey

Presentation of Final Projects

Groups:

Presentation of Final Projects

Semester & Mini-2 Last Day of Classes = Friday, December 9, 2022

Final Written Reports for Final Projects due Monday, 12/12/2022 at 11:59pm.

Final Test: Start Tuesday, 12/13/2022 at 12:01am ET; Due Wednesday, 12/14/2022 at 11:59pm ET (all day Tuesday and Wednesday)

Final Grades due: December 21, 2022 by 4 pm