Demonstrational Interfaces

Report on an NSF Funded Research,
October, 1995

Brad A. Myers
Human Computer Interaction Institute
Carnegie Mellon University


For more Information:

See the Web page of the Demonstrational Interfaces Group


CONTACT INFORMATION:

Brad A. Myers
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213-3891
(412) 268-5150
FAX: (412) 268-5576
bam@cs.cmu.edu


PROGRAM AREA:

5. Usability and User-Centered Design.

KEYWORDS:

Programming by Demonstration (PBD), End-User Programming (EUP), Demonstrational Interfaces, Visualization, Visual Programming

PROJECT SUMMARY:

Demonstrational interfaces allow the user to give examples in the normal interface of an application and the system generalizes from the examples to create parameterized procedures or relationships. By inferring generalizations from the examples, the system provides the ease-of-use of direct manipulation while extending the range of what can be accomplished.

There are four fundamental research questions we have been exploring. The most important is how the users will understand and correct the inferences made by the system. Any system that generalizes from examples is occasionally going to guess wrong, and it is important that users know what the system is doing so they will feel comfortable and in control. The PURSUIT visual shell (an iconic interface to a file system, like the Macintosh finder) incorporates our most successful and novel mechanism for this feedback. PURSUIT is the PhD thesis of my student, Francesmary Modugno, and was just completed in May, 1995. As the user demonstrates a program by executing example commands, PURSUIT builds a visual language representation of operations and inferences. The visual language is based on the "comic strip metaphor": the panels show the relevant data, and changes from one panel to the next represent the operation. The language provided a single medium for verifying and correcting inferences, for reviewing completed programs later, and for editing programs. Despite skepticism from other researchers that the interface would be usable, formal user studies showed that nonprogrammers could create fairly complex programs, and that the visual language was more effective than an equivalent textual language.

The second fundamental research question for demonstrational interfaces is the appropriate representations and inferencing algorithms. For a system to be acceptable, it must guess right most of the time. Our early systems used straightforward rule-based techniques and pattern matching that were empirically tuned to give acceptable performance. These resulted in highly predictable user interfaces, and this technique has been adopted by most other demonstrational systems. We are now beginning to explore a more elaborate algorithm to provide powerful inferences that can take into account the user's hints.

The third research question is what architectural support can be supplied in a toolkit to make demonstrational interfaces easier to build. Many forms of demonstrational interfaces record a sequence of operations and allow them to be repeated. We have developed an architecture where programmers structure applications around "hierarchical command objects," which encapsulate the information needed to execute, undo and re-execute each operation. They are hierarchical to support structural decomposition of user interface behaviors.

The fourth research question is to which domains can demonstrational interfaces be successfully applied. We have identified useful aspects of user interface construction, text formatting (funded by a previous NSF grant), file manipulation in a visual shell (in Pursuit discussed above), data visualization, and educational game construction, that are appropriate for being demonstrational. They share the properties that the natural way a person would describe the problem to another person is by drawing examples, and that domain knowledge can be used to narrow the range of possibilities for generalizing from the examples. Each domain also illuminates new issues for feedback and representations.


PROJECT REFERENCES:

The PURSUIT Visual Shell

Francesmary Modugno. Extending End-User Programming in a Visual Shell with Programming by Demonstration and Graphical Language Techniques. PhD Thesis. Computer Science Department, Carnegie Mellon University Technical Report CMU-CS-95-130. 1995.

Francesmary Modugno, Albert T. Corbett and Brad A. Myers. "Evaluating Program Representation in a Demonstrational Visual Shell." Experimental Studies of Programmers Sixth Workshop. Jan. 5-7, 1996. Alexandria, VA. To appear. Abstract, Postscript

Francesmary Modugno and Brad A. Myers. "A State-Based Visual Language for a Demonstrational Visual Shell," 1994 IEEE Workshop on Visual Languages. St. Louis, MO. pp. 304-311. Abstract and Postscript

Francesmary Modugno and Brad A. Myers. "Pursuit: A Demonstrational Visual Shell," Technical Video Program of the CHI'94 conference. SIGGRAPH Video Review, Issue 97, no. 12.

Francesmary Modugno, T.R.G. Green and Brad A. Myers. "Visual Programming in a Visual Domain: A Case Study of Cognitive Dimension," Human-Computer Interaction '94, People and Computers. Glasgow, Scotland, August, 1994. pp. 91-108. Abstract and Postscript

Francesmary Modugno and Brad A. Myers. "Exploring Graphical Feedback in a Demonstrational Visual Shell," The 1994 East-West International Conference on Human-Computer Interaction (EWHCI'94). St. Petersburg, Russia, August, 1994. pp. 262-272. An updated version appears in Lecture Notes in Computer Science 876. Brad Blumenthal, Juri Gornostaev and Claus Unger, Editors. Springer-Verlag, 1994. Abstract and Postscript

Data Visualization:

Brad A. Myers, Jade Goldstein, and Matthew A. Goldberg. "Creating Charts by Demonstration," Proceedings CHI'94: Human Factors in Computing Systems. Boston, MA, Apr. 24-28, 1994. pp. 106-111.

A new command architecture to support PBD:

Brad A. Myers and David Kosbie. "Reusable Hierarchical Command Objects." Proceedings CHI'96: Human Factors in Computing Systems. Vancouver, BC, Canada. April 14-18, 1996. To Appear. Postscript

David S. Kosbie and Brad A. Myers, "Extending Programming By Demonstration With Hierarchical Event Histories," The 1994 East-West International Conference on Human-Computer Interaction. St. Petersburg, Russia, August, 1994. pp. 147-157. Abstract and Postscript

David S. Kosbie and Brad A. Myers. "A System-Wide Macro Facility Based on Aggregate Events: A Proposal," Watch What I Do: Programming by Demonstration, Allen Cypher, et. al., eds. Cambridge, MA: The MIT Press, 1993. pp. 433-444.

Demonstrational Text Formatting:

Andrew Werth, Tourmaline: Formatting Document Headings by Example. Carnegie Mellon University Information Networking Institute MS Thesis, 1992.

Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Proceedings INTERCHI'93: Human Factors in Computing Systems. Amsterdam, The Netherlands, April 24-29, 1993. p. 532.

Andrew J. Werth and Brad A. Myers. "Tourmaline: Macrostyles by Example," Technical Video Program, INTERCHI'93 conference. Amsterdam, The Netherlands, April, 1993. SIGGRAPH Video Review, Issue 89, no. 17.

Brad A. Myers. "Text Formatting by Demonstration," Proceedings SIGCHI'91: Human Factors in Computing Systems. New Orleans, LA. April 28-May 2, 1991. pp. 251-256.


AREA BACKGROUND:

A "demonstrational interface" watches while the user executes conventional direct manipulation actions, but creates a more general abstraction from the specific example. For instance, suppose a user wants to delete all the ".ps" files from a directory. The user could drag a file named "v1.ps" to the trash can, and then a file named "v2.ps", and a demonstrational system might automatically create a macro to delete the rest.

The motivation behind demonstrational interfaces is simple and compelling: if a user knows how to perform a task on the computer, that should be sufficient to create a program to perform a the task or a generalization of that task. The goal is to give the power of programming to all computer users without requiring them to learn how to program.


AREA REFERENCES:

Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers and Alan Turransky, eds. Watch What I Do: Programming by Demonstration. Cambridge, MA: The MIT Press, 1993.

Brad A. Myers. "Demonstrational Interfaces: A Step Beyond Direct Manipulation," IEEE Computer. August, 1992. vol. 25, no. 8. pp. 61-73.

Brad A. Myers. "Invisible Programming," 1990 IEEE Workshop on Visual Languages. Skokie, Ill, October 4-6, 1990. pp. 203-208.


RELATED PROGRAM AREAS:

4. Adaptive Human Interfaces

Visualization and Interactive Computing.

Intelligent Interfaces.


POTENTIAL RELATED PROJECTS:

My Garnet and Amulet User Interface Development Environments provide demonstrational interfaces on many interactive tools. Furthermore, these environments are good for creating demonstrational interfaces for other systems. A number of NSF Interactive Systems principal investigators are using or collaborating on these.

Amulet in particular is being designed with the specific goal of supporting user interface researchers, so we hope that it will prove useful to NSF ISP.


Last modified December 7, 1995 by Brad Myers