User Interfaces that Span Hand-Held
and Fixed Devices

Position paper for the CHI'2001 Workshop on
Distributed and Disappearing User Interfaces in Ubiquitous Computing

Brad A. Myers
Jeff Nichols
Rob Miller

Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213-3891
(412) 268-5150
FAX: (412) 268-1266
bam+@cs.cmu.edu
http://www.cs.cmu.edu/~bam

The Pebbles research project (http://www.pebbles.cs.cmu.edu/) has been studying the use of hand-held devices at the same time as other computing devices.  As people move about the world, they will be entering and leaving spaces where there are embedded or desktop computers, such as offices, conference rooms, classrooms, and even "smart homes." We are exploring the many issues surrounding how to have the user interface, functionality, and information spread across multiple devices that are used simultaneously. For example, there are many ways that a personal digital assistant (PDA), such as a Palm Pilot or PocketPC device, can serve as a useful adjunct to a personal computer to enhance interactions with existing desktop applications for individuals or groups. Applications may distribute their user interfaces across multiple devices so the user can choose the appropriate device for each part of the interaction.  A key focus of our research is that the hand-held computers are used both as output devices and as input devices to control the activities on the other computers. The following scenarios illustrate some of the capabilities we are already investigating:

There are many significant research issues involved in bringing these visions to fruition, which we are investigating. We are particularly interested in the appropriate ways to distribute the user interface across multiple devices, how to support multiple people interacting with the same screen using their personal devices as auxiliary input and output devices, and usability issues with multi-device interaction techniques.

The Pebbles research project has made substantial progress by building example applications, releasing them for general use, and formally testing them in usability experiments. Several of our existing applications support meetings where the participants are co-located. All participants' PDAs are in continuous two-way communication with the main computer which is often projected on a screen to serve as the focal point of the discussion. Some of our initial applications use the PDAs as remote mice and keyboards so that everyone in the meeting can control the main computer. Other applications include controlling a PowerPoint presentation while displaying the slide notes and titles on the PDA, as a shared whiteboard that supports multiple inputs simultaneously, for private side messages via a "chat" program, and to display multiple cursors for pointing and scribbling on arbitrary applications [Myers 1998]. We are currently investigating a number of groupware issues, including appropriate floor control mechanisms, and how to fluidly move information between the public and private displays. Another set of applications supports a single person using the PDA as an extra input and output device to enhance desktop applications. The PDA can be used as a scrolling device, as a general-purpose button panel (to create screens of "shortcuts"), as an index page or table of contents for web surfing, and to cut and paste information back and forth from the PDA to the PC [Myers 2001]. These applications have been downloaded over 20,000 times, and are available from http://www.pebbles.cs.cmu.edu/. One application has been licensed and is now available for commercial sale: http://www.slideshowcommander.com/.

Brad Myers, the principal investigator leading the Pebbles research project, would like to participate in this workshop to discuss various visions of the future of user interfaces in ubiquitous computing. Our research group believes that there is not enough research about the issues around carrying hand-held devices in and out of spaces where other hand-held and fixed devices are operating. Viewing the handhelds as part of an integrated user interface system can have a significant impact on the user interfaces, communication mechanisms, hardware, and operating system of the devices. From this workshop's title, we feel that clearly, the interfaces will be distributed across multiple devices and modalities. Although one might hope the interfaces will disappear into the environment, we believe that there will always be devices that the users bring with them, and that the interfaces on these devices and the interfaces that span these devices and the environments are important to study. (Jeff Nichols, a PhD student working with Brad Myers on this project, is proposing to attend the other ubiquitous computing CHI'2001 workshop #10, but might be able to come to this workshop on Sunday.)

Short Biography of Brad Myers

Brad A. Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects including the Pebbles PDA Project, the User Interface Software Project, Demonstrational Interfaces, and the Natural Programming Project. He is the author or editor of over 200 publications, including the books Creating User Interfaces by Demonstration and Languages for Developing User Interfaces, and he is on the editorial board of five journals. He has been a consultant on user interface design and implementation to 40 companies. Myers received a PhD in computer science at the University of Toronto where he developed the Peridot UIMS. He received the MS and BSc degrees from the Massachusetts Institute of Technology during which time he was a research intern at Xerox PARC. From 1980 until 1983, he worked at PERQ Systems Corporation. His research interests include Mobile Devices, User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility.

References

[Myers 1998] Myers, B.A., Stiel, H., and Gargiulo, R. “Collaboration Using Multiple PDAs Connected to a PC,” in Proceedings CSCW'98: ACM Conference on Computer-Supported Cooperative Work. 1998. Seattle, WA: pp. 285-294.

[Myers 2000] Myers, B.A., Lie, K.P.L., and Yang, B.-C.J. “Two-Handed Input Using a PDA And a Mouse,” in Proceedings CHI'2000: Human Factors in Computing Systems. April, 2000. The Hague, The Netherlands: pp. 41-48

[Myers 2001] Brad A. Myers, Robert C. Miller, Benjamin Bostwick, and Carl Evankovich, "Extending the Windows Desktop Interface With Connected Handheld Computers," 4th USENIX Windows Systems Symposium, August 3-4, 2000, Seattle, WA. pp. 79-88. Adobe Acrobat (pdf).

[Nichols 2001] Jeffrey W. Nichols. "Using Handhelds as Controls for Everyday Appliances: A Paper Prototype Study." ACM CHI'2001 Student Posters. Seattle, WA. March 31-April 5, 2001. To appear. Adobe Acrobat (pdf).