Human Computer Interaction Institute
Carnegie Mellon University
There are four fundamental research questions we have been exploring. The most important is how the users will understand and correct the inferences made by the system. Any system that generalizes from examples is occasionally going to guess wrong, and it is important that users know what the system is doing so they will feel comfortable and in control. The PURSUIT visual shell (an iconic interface to a file system, like the Macintosh finder) incorporates our most successful and novel mechanism for this feedback. PURSUIT is the PhD thesis of my student, Francesmary Modugno, and was completed in May, 1995. As the user demonstrates a program by executing example commands, PURSUIT builds a visual language representation of operations and inferences. The visual language is based on the "comic strip metaphor:" the panels show the relevant data, and changes from one panel to the next represent the operation. The language provides a single medium for verifying and correcting inferences, for reviewing completed programs later, and for editing programs. Despite skepticism from other researchers that the interface would be usable, formal user studies showed that nonprogrammers could create fairly complex programs, and that the visual language was more effective than an equivalent textual language.
The second fundamental research question for demonstrational interfaces is the appropriate representations and inferencing algorithms. For a system to be acceptable, it must guess right most of the time. Our early systems used straightforward rule-based techniques and pattern matching that were empirically tuned to give acceptable performance. These resulted in highly predictable user interfaces, and this technique has been adopted by most other demonstrational systems. We are now beginning to explore a more elaborate algorithm to provide powerful inferences that can take into account the user's hints.
The third research question is what architectural support can be supplied in a toolkit to make demonstrational interfaces easier to build. Many forms of demonstrational interfaces record a sequence of operations and allow them to be repeated. We have developed an architecture where programmers structure applications around "hierarchical command objects," which encapsulate the information needed to execute, undo and re-execute each operation. They are hierarchical to support structural decomposition of user interface behaviors.
The fourth research question is to which domains can demonstrational interfaces be successfully applied. We have identified useful aspects that are appropriate for using demonstrational user interfaces in applications as varied as user interface construction, text formatting (funded by a previous NSF grant), file manipulation in a visual shell (in Pursuit discussed above), data visualization, educational game construction, and creating dynamic world-wide-web pages. They share the properties that the natural way a person would describe the problem to another person is by drawing examples, and that domain knowledge can be used to narrow the range of possibilities for generalizing from the examples. Each domain also illuminates new issues for feedback and representations.
Francesmary Modugno, Albert T. Corbett, and Brad A. Myers. "Graphical Representation of Programs in a Demonstrational Visual Shell -- An Empirical Evaluation," ACM Transactions on Computer-Human Interaction. To appear, 1998.
Francesmary Modugno and Brad A. Myers. "Visual Programming in a Visual Shell -- A Unified Approach," Journal of Visual Languages and Computing, to appear. (Postscript)
Francesmary Modugno, Albert T. Corbett, and Brad A. Myers. "Evaluating Program Representations in a Demonstrational Visual Shell," CHI'95 Conference Companion: Human Factors in Computing Systems. Denver, CO. May, 1995. pp. 234-235.
Francesmary Modugno and Brad A. Myers. "A State-Based Visual Language for a Demonstrational Visual Shell," 1994 IEEE Workshop on Visual Languages. St. Louis, MO. pp. 304-311. Abstract and Postscript
Francesmary Modugno and Brad A. Myers. "Pursuit: A Demonstrational Visual Shell," Technical Video Program of the CHI'94 conference. SIGGRAPH Video Review, Issue 97, no. 12.
Francesmary Modugno and Brad A. Myers. "Pursuit: Graphically Representing Programs in a Demonstrational Visual Shell," CHI'94 Conference Companion. Boston, MA, Apr. 24-28, 1994. pp. 455-456.
Francesmary Modugno, T.R.G. Green and Brad A. Myers. "Visual Programming in a Visual Domain: A Case Study of Cognitive Dimension," Human-Computer Interaction '94, People and Computers. Glasgow, Scotland, August, 1994. pp. 91-108. Abstract and Postscript
Francesmary Modugno and Brad A. Myers. "Exploring Graphical Feedback in a Demonstrational Visual Shell," The 1994 East-West International Conference on Human-Computer Interaction (EWHCI'94). St. Petersburg, Russia, August, 1994. pp. 262-272. An updated version appears in Lecture Notes in Computer Science 876. Brad Blumenthal, Juri Gornostaev and Claus Unger, Editors. Springer-Verlag, 1994. Abstract and Postscript
Francesmary Modugno and Brad A. Myers. "Graphical Representation and Feedback in a PBD System," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 423-431.
Brad A. Myers, Jade Goldstein, and Matthew A. Goldberg. Creating Charts and Visualizations by Demonstration. Patent Number 5,581,677. Filed April 22, 1994.
David S. Kosbie and Brad A. Myers, "Extending Programming By Demonstration With Hierarchical Event Histories," The 1994 East-West International Conference on Human-Computer Interaction. St. Petersburg, Russia, August, 1994. pp. 147-157. Abstract and Postscript
David S. Kosbie and Brad A. Myers. "A System-Wide Macro Facility Based on Aggregate Events: A Proposal," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 433-444.
Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Proceedings INTERCHI'93: Human Factors in Computing Systems. Amsterdam, The Netherlands, April 24-29, 1993. p. 532.
Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Technical Video Program, INTERCHI'93 conference. Amsterdam, The Netherlands, April, 1993. SIGGRAPH Video Review, Issue 89, no. 17.
Brad A. Myers. "Text Formatting by Demonstration," Proceedings SIGCHI'91: Human Factors in Computing Systems. New Orleans, LA. April 28-May 2, 1991. pp. 251-256.
Brad A. Myers. "Tourmaline: Text Formatting by Demonstration," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 309-321.
Richard G. McDaniel and Brad A. Myers. Improving Demonstration Using Better Interaction Techniques. Carnegie Mellon University School of Computer Science Technical Report CMU-CS-97-103 and Human Computer Interaction Institute Technical Report CMU-HCII-97-100, January, 1997. Postscript.
The motivation behind demonstrational interfaces is simple and compelling: if a user knows how to perform a task on the computer, that should be sufficient to create a program to perform a the task or a generalization of that task. The goal is to give the power of programming to all computer users without requiring them to learn how to program.
Brad A. Myers. "Demonstrational Interfaces: A Step Beyond Direct Manipulation," IEEE Computer. August, 1992. vol. 25, no. 8. pp. 61-73.
Brad A. Myers. "Invisible Programming," 1990 IEEE Workshop on Visual Languages. Skokie, Ill, October 4-6, 1990. pp. 203-208.