Using a Hand-Held as a
Personal Universal Controller
The Pebbles project is investigating how a hand-held
computer can be used as a "Personal Universal Controller".
Personal digital assistants (PDAs) like the
PocketPC and Palm
Pilot are becoming increasingly ubiquitous, and with wireless technologies
such as BlueTooth and IEEE 802.11, they will be in close interactive
communication with other devices. Furthermore, cell-phones and pagers,
which primarily used for communication, are increasingly becoming
programmable. We are investigating how these kinds of hand-held
devices devices can be used to control all kinds of
home, office and factory equipment. The concept is that when users point
their own hand-held at a light switch, at a photocopier in an office, at a
machine tool in a factory, at a VCR at home, at a piece of test equipment
in the field, or at almost any other kind of device, the device will send
to the hand-held a description of its control parameters. The hand-held
uses this information to create an appropriate control panel, taking into
account the properties of the controls that are needed, the properties of
the hand-held (the display type and input techniques available), and the
properties of the user (what language is preferred, whether left or right
handed, how big the buttons should be based on whether the user prefers
using a finger or a stylus). The user can then control the device using
the hand-held. The device will not need to dedicate much processing power,
hardware, or cost to the user interface, since it will only need to
contain a description of its capabilities and storage for the current
settings, along with hardware for wireless communication. The hand-held
programs will use intelligent "model-based" techniques to create
useful and appropriate interfaces that are customized for each user.
Furthermore, though a collaboration with the Universal
Speech Interfaces project at CMU, we are demonstrating that the same
specification can be used to generate a speech interface, and then the
appliance can be controlled using speech and/or a graphical user interface.
In our preliminary study, using the Palm
and PocketPC interfaces shown below were twice as fast and resulted in one-fifth
the errors compared to the manufacturer's interface.
See the publications below.
Note that you cannot use any of
the Pebbles software to control real applications (other than a PC)
at this time. See Links section below.
A key part of our vision is the description of
the control parameters that the controller uses for
automatically creating control panels. We have developed a
language for specifying these control parameters, and a
protocol for transmitting changes to the state of these
parameters between the appliance and the controller. This
language and the protocol are based on the XML standard. You may
be interested in the documentation and DTDs for these
Part of this research was performed in collaboration with Maya Design.
CMU people on the PUC project include:
- Prof. Jodi Forlizzi
- Pegeen Shen
- Kevin Litwack
- Rajesh Seenichamy
- Suporn Pongnumkul
- Mathilde Pignol
- Marc Khadpe
The complete specifications for the PUC protocols are
available. See specification language section above.
- Talk by CMU and Maya on progress to date presented
to the Pittsburgh Digital Greenhouse, February 13, 2002. PowerPoint
- Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joe
Hughes, Thomas K. Harris, Roni Rosenfeld, Mathilde Pignol. "Generating
Remote Control Interfaces for Complex Appliances." CHI Letters: ACM Symposium on User Interface Software and
Technology, UIST'02, 27-30 Oct. 2002, Paris, France. To appear.
Jeffrey Nichols, Brad Myers, Thomas K. Harris, Roni Rosenfeld,
Stefanie Shriver, Michael Higgins, Joseph Hughes. "Requirements for
Automatically Generating Multi-Modal Interfaces for Complex
IEEE Fourth International Conference on Multimodal Interfaces,
Pittsburgh, PA. October 14-16, 2002. pp. 377-382.
- Jeffrey W. Nichols. "Using Handhelds as
Controls for Everyday Appliances: A Paper Prototype Study." ACM
CHI'2001 Student Posters. Seattle, WA. March 31-April 5, 2001. pp.
Brad A. Myers, Jeff Nichols, Rob Miller.
"User Interfaces that Span Hand-Held and Fixed Devices"
Workshop on Distributed and Disappearing User Interfaces in
Ubiquitous Computing at CHI'2001, Seattle, WA.
Albrecht Schmidt, Peter Ljundgstrand, and Anind Dey, editors.
University of Karlsruhe Faculty of Information Technical Report
2001-6. ISSN 1432-7864.
Jeffrey Nichols, Brad A. Myers, Rob Miller. "Personal Interfaces in
Ubiquitous Environments". Workshop on Building the Ubiquitous
Computing User Experience at CHI'2001, Seattle, WA.
Brad A. Myers and Jeffrey Nichols, "Communication Ubiquity
Enables Ubiquitous Control." 'Boaster' for Human-Computer
Interaction Consortium (HCIC'2002). Winter Park, CO, Feb, 2002. html
Jeffrey Nichols. "Informing Automatic Generation of Remote Control
Interfaces with Human Designs" CHI'2002 Student Posters. Minneapolis, Minnesota, April 20-25, 2002.
pp. 864-865. Adobe
Article about this research:
Michael Yeomans, "CMU Scientists Improving Computers' People
Skills," Pittsburgh Tribune-Review, Tuesday, October 22,
2002. pages B7, B10.
Mike Crissey, "Designers Work on All-in-One Remote,"
Associated Press, August 27, 2002.
Funding for the use of Pebbles as a
Personal Universal Controller is provided by:
are investigating various ways to connect our PUC to appliances, including
Note: the PUC system cannot be used to control real
appliances today. Some systems for using handhelds as remote controls
- Philips Pronto
- OmniRemote Pro by Pacific Neo-Tek, Inc. for Palm
- NoviiRemote by NoviiMedia for Palm and PocketPC
- VITO Remote by VITO Technology for PocketPC
- UniversalRemoteCE by Pyramid Peak Design for PocketPC