Demonstrational Interfaces

Brad A. Myers

Human Computer Interaction Institute
Carnegie Mellon University

CONTACT INFORMATION

Brad A. Myers
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213-3891
(412) 268-5150
FAX: (412) 268-5576
bam@cs.cmu.edu

WWW PAGE

http://www.cs.cmu.edu/~bydemo

PROGRAM AREA

5. Usability and User-Centered Design.

KEYWORDS

Programming by Demonstration (PBD), End-User Programming (EUP), Demonstrational Interfaces, Visualization, Visual Programming

PROJECT SUMMARY

Demonstrational interfaces allow the user to give examples in the normal interface of an application and the system generalizes from the examples to create parameterized procedures or relationships. By inferring generalizations from the examples, the system provides the ease-of-use of direct manipulation while extending the range of what can be accomplished.

There are four fundamental research questions we have been exploring. The most important is how the users will understand and correct the inferences made by the system. Any system that generalizes from examples is occasionally going to guess wrong, and it is important that users know what the system is doing so they will feel comfortable and in control. The PURSUIT visual shell (an iconic interface to a file system, like the Macintosh finder) incorporates our most successful and novel mechanism for this feedback. PURSUIT is the PhD thesis of my student, Francesmary Modugno, and was completed in May, 1995. As the user demonstrates a program by executing example commands, PURSUIT builds a visual language representation of operations and inferences. The visual language is based on the "comic strip metaphor:" the panels show the relevant data, and changes from one panel to the next represent the operation. The language provides a single medium for verifying and correcting inferences, for reviewing completed programs later, and for editing programs. Despite skepticism from other researchers that the interface would be usable, formal user studies showed that nonprogrammers could create fairly complex programs, and that the visual language was more effective than an equivalent textual language.

The second fundamental research question for demonstrational interfaces is the appropriate representations and inferencing algorithms. For a system to be acceptable, it must guess right most of the time. Our early systems used straightforward rule-based techniques and pattern matching that were empirically tuned to give acceptable performance. These resulted in highly predictable user interfaces, and this technique has been adopted by most other demonstrational systems. We are now beginning to explore a more elaborate algorithm to provide powerful inferences that can take into account the user's hints.

The third research question is what architectural support can be supplied in a toolkit to make demonstrational interfaces easier to build. Many forms of demonstrational interfaces record a sequence of operations and allow them to be repeated. We have developed an architecture where programmers structure applications around "hierarchical command objects," which encapsulate the information needed to execute, undo and re-execute each operation. They are hierarchical to support structural decomposition of user interface behaviors.

The fourth research question is to which domains can demonstrational interfaces be successfully applied. We have identified useful aspects that are appropriate for using demonstrational user interfaces in applications as varied as user interface construction, text formatting (funded by a previous NSF grant), file manipulation in a visual shell (in Pursuit discussed above), data visualization, educational game construction, and creating dynamic world-wide-web pages. They share the properties that the natural way a person would describe the problem to another person is by drawing examples, and that domain knowledge can be used to narrow the range of possibilities for generalizing from the examples. Each domain also illuminates new issues for feedback and representations.

PROJECT REFERENCES

Overview of the Project

Brad A. Myers, Francesmary Modugno, Rich McDaniel, David Kosbie, Andrew Werth, Robert C. Miller, John Pane, James Landay, Jade Goldstein, and Matthew A. Goldberg, "The Demonstrational Interfaces Project at CMU," 1996 AAAI Spring Symposium on Acquisition, Learning and Demonstration: Automating Tasks for Users. March 25-27, 1996, Stanford, CA. Technical Report SS-96-02, pp. 85-91. Postscript.

The PURSUIT Visual Shell

Francesmary Modugno. Extending End-User Programming in a Visual Shell with Programming by Demonstration and Graphical Language Techniques. PhD Thesis. Computer Science Department, Carnegie Mellon University Technical Report CMU-CS-95-130. 1995.

Francesmary Modugno, Albert T. Corbett, and Brad A. Myers. "Graphical Representation of Programs in a Demonstrational Visual Shell -- An Empirical Evaluation," ACM Transactions on Computer-Human Interaction. To appear, 1998.

Francesmary Modugno and Brad A. Myers. "Visual Programming in a Visual Shell -- A Unified Approach," Journal of Visual Languages and Computing, to appear. (Postscript)

Francesmary Modugno, Albert T. Corbett, and Brad A. Myers. "Evaluating Program Representations in a Demonstrational Visual Shell," CHI'95 Conference Companion: Human Factors in Computing Systems. Denver, CO. May, 1995. pp. 234-235.

Francesmary Modugno and Brad A. Myers. "A State-Based Visual Language for a Demonstrational Visual Shell," 1994 IEEE Workshop on Visual Languages. St. Louis, MO. pp. 304-311. Abstract and Postscript

Francesmary Modugno and Brad A. Myers. "Pursuit: A Demonstrational Visual Shell," Technical Video Program of the CHI'94 conference. SIGGRAPH Video Review, Issue 97, no. 12.

Francesmary Modugno and Brad A. Myers. "Pursuit: Graphically Representing Programs in a Demonstrational Visual Shell," CHI'94 Conference Companion. Boston, MA, Apr. 24-28, 1994. pp. 455-456.

Francesmary Modugno, T.R.G. Green and Brad A. Myers. "Visual Programming in a Visual Domain: A Case Study of Cognitive Dimension," Human-Computer Interaction '94, People and Computers. Glasgow, Scotland, August, 1994. pp. 91-108. Abstract and Postscript

Francesmary Modugno and Brad A. Myers. "Exploring Graphical Feedback in a Demonstrational Visual Shell," The 1994 East-West International Conference on Human-Computer Interaction (EWHCI'94). St. Petersburg, Russia, August, 1994. pp. 262-272. An updated version appears in Lecture Notes in Computer Science 876. Brad Blumenthal, Juri Gornostaev and Claus Unger, Editors. Springer-Verlag, 1994. Abstract and Postscript

Francesmary Modugno and Brad A. Myers. "Graphical Representation and Feedback in a PBD System," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 423-431.

Data Visualization:

Brad A. Myers, Jade Goldstein, and Matthew A. Goldberg. "Creating Charts by Demonstration," Proceedings CHI'94: Human Factors in Computing Systems. Boston, MA, Apr. 24-28, 1994. pp. 106-111.

Brad A. Myers, Jade Goldstein, and Matthew A. Goldberg. Creating Charts and Visualizations by Demonstration. Patent Number 5,581,677. Filed April 22, 1994.

A new command architecture to support PBD:

Brad A. Myers and David Kosbie. "Reusable Hierarchical Command Objects." Proceedings CHI'96: Human Factors in Computing Systems. Vancouver, BC, Canada. April 14-18, 1996. 260-267. html version or another place for the html. postscript

David S. Kosbie and Brad A. Myers, "Extending Programming By Demonstration With Hierarchical Event Histories," The 1994 East-West International Conference on Human-Computer Interaction. St. Petersburg, Russia, August, 1994. pp. 147-157. Abstract and Postscript

David S. Kosbie and Brad A. Myers. "A System-Wide Macro Facility Based on Aggregate Events: A Proposal," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 433-444.

Demonstrational Text Formatting:

Andrew Werth, Tourmaline: Formatting Document Headings by Example. Carnegie Mellon University Information Networking Institute MS Thesis, 1992.

Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Proceedings INTERCHI'93: Human Factors in Computing Systems. Amsterdam, The Netherlands, April 24-29, 1993. p. 532.

Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Technical Video Program, INTERCHI'93 conference. Amsterdam, The Netherlands, April, 1993. SIGGRAPH Video Review, Issue 89, no. 17.

Brad A. Myers. "Text Formatting by Demonstration," Proceedings SIGCHI'91: Human Factors in Computing Systems. New Orleans, LA. April 28-May 2, 1991. pp. 251-256.

Brad A. Myers. "Tourmaline: Text Formatting by Demonstration," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 309-321.

Demonstrational Interfaces for the World Wide Web

Robert C. Miller, Brad A. Myers. Creating Dynamic World Wide Web Pages by Demonstration. Carnegie Mellon University School of Computer Science Technical Report, no. CMU-CS-97-131 and Human Computer Interaction Institute Technical Report CMU-HCII-97-101. May, 1997. Postscript

Demonstrational Interfaces for Building Software

Brad Vander Zanden and Brad A. Myers. "Demonstrational and Constraint-Based Techniques for Pictorially Specifying Application Objects and Behaviors," ACM Transactions on Computer-Human Interaction. Vol. 2, no. 4, Dec, 1995. pp. 308-356.

Richard G. McDaniel and Brad A. Myers. Improving Demonstration Using Better Interaction Techniques. Carnegie Mellon University School of Computer Science Technical Report CMU-CS-97-103 and Human Computer Interaction Institute Technical Report CMU-HCII-97-100, January, 1997. Postscript.

AREA BACKGROUND

A "demonstrational interface" watches while the user executes conventional direct manipulation actions, but creates a more general abstraction from the specific example. For instance, suppose a user wants to delete all the ".ps" files from a directory. The user could drag a file named "v1.ps" to the trash can, and then a file named "v2.ps", and a demonstrational system might automatically create a macro to delete the rest.

The motivation behind demonstrational interfaces is simple and compelling: if a user knows how to perform a task on the computer, that should be sufficient to create a program to perform a the task or a generalization of that task. The goal is to give the power of programming to all computer users without requiring them to learn how to program.

AREA REFERENCES

Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers and Alan Turransky, eds. Watch What I Do: Programming by Demonstration. Cambridge, MA: The MIT Press, 1993.

Brad A. Myers. "Demonstrational Interfaces: A Step Beyond Direct Manipulation," IEEE Computer. August, 1992. vol. 25, no. 8. pp. 61-73.

Brad A. Myers. "Invisible Programming," 1990 IEEE Workshop on Visual Languages. Skokie, Ill, October 4-6, 1990. pp. 203-208.

RELATED PROGRAM AREAS

4. Adaptive Human Interfaces

POTENTIAL RELATED PROJECTS

My Amulet User Interface Development Environment provides demonstrational interfaces on many interactive tools. Furthermore, Amulet is specifically being designed to support user interface researchers, so we hope that it will prove useful to the NSF ISP in general. Over 30 research projects all over the world are already using Amulet, including a number of NSF Interactive Systems principal investigators, and some of these are demonstrational systems.