Communication Ubiquity Enables Ubiquitous Control

"Boaster" for Human-Computer Interaction Consortium (HCIC'2002)

Brad A. Myers and Jeffrey Nichols

Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
5000 Forbes Avenue
Pittsburgh, PA 15213-3891
(412) 268-5150
FAX: (412) 268-1266

November 27, 2001

Short Abstract:

Handheld devices, such as cell-phones and PDAs, can be used for more than just communication. What happens when every person has a handheld that ubiquitously communicates with every appliance in the environment? As part of the Pebbles project, we are exploring technologies that allow handhelds to monitor and control appliances.

Extended Abstract:

Everyday appliances, including telephones, copiers, home stereos, light switches, etc., increasingly contain embedded computers which enable greater functionality. If the interfaces to these appliances were easy to use, people might benefit from these new functions. Unfortunately, it is rare to find a well-designed appliance interface [Brouwer-Janse 1992].

We are investigating how an intelligent hand-held Personal Universal Controller (PUC) can be used to make appliances easier to control as part of the Pebbles research project [Myers 2001]. The concept is that people can use their own handheld to control all appliances and devices in their environment, such as the lights, a photocopier in an office, a machine tool in a factory, a VCR at home, a piece of test equipment in the field, etc. The appliance sends the handheld a description of its control parameters.  The handheld uses this information to automatically create an appropriate control panel, taking into account the properties of the handheld (the display type and input techniques available) and the properties of the user (what language is preferred, whether the user is left or right handed, how big the buttons should be based on whether the user prefers using a finger or a stylus). The user can then control the appliance using the handheld. Unlike today's universal remotes, the PUC will be self-programming and use two-way communication: it will create control panels for new devices without user intervention, and it will receive feedback from the appliances to help keep the user informed as to the device state.

We believe that the PUC can address many of the difficult problems inherent in ubiquitous technologies. Here are some examples:

There are many significant research issues involved in bringing this vision to life, and we are in the process of investigating many of them. Our initial focus is on interface generation and the specification language that supports it. We began our research by creating hand-designed interfaces for an AIWA shelf stereo system and an AT&T office telephone/answering machine, targeted both at the Palm and PocketPC hand-held computers. The purpose of the hand-designed interfaces is two-fold: 

1)     To verify that better remote control interfaces can be created on a hand-held.

2)      To understand what functional knowledge about an appliance will be needed to construct a high-quality remote control interface.

We performed two user studies to test the first point. Results showed that users made about 1/5 the errors and took about 1/2 the time to perform complex tasks using the handheld interfaces compared to the actual manufacturer's interfaces for the stereo and telephone [Nichols 2001]. We have also gained knowledge that is needed to create high-quality remote control interfaces. For example, we have noticed that the layout problem for many remote control interfaces can be greatly reduced by grouping together related items. These groups can often be reduced to having a small number of items, which are then easy to place on the screen relative to each other. Furthermore, these groupings can often be inferred from knowledge about when a function can be used. For example, in the stereo interface shown in the figure below, it is possible to infer that Tape and CD functions should be in separate panels because their functions cannot be used at the same time.

Now, we are using the knowledge gained from evaluating our hand-designed interfaces to create a specification language and an automatic generation and layout algorithm that will create similar interfaces. The challenge is to make the specification flexible enough to support a wide-range of form factors and interface modalities, while still providing high-quality user interfaces. As a part of this, we are working with Carnegie Mellon’s Universal Speech Interface Group to investigate the possibilities of using our specification as the basis for speech interfaces. We are also exploring how to connect to real appliances. For example, we are collaborating with Premise Systems to look at using the PUC for home automation control. There is hope that standards such as Universal Plug-And-Play (uPnP), BlueTooth, Home Audio Video Interoperability (HAVi), or other companies' proprietary protocols will provide interactive two-way control of appliances in the future. Sony, for example, has set a goal of connecting most of their products to the network by 2005 [Sony 2001].

For the future, there are many technological, user interface, and cultural issues that will be important to study. There are questions of how to automatically generate remote control interfaces and make them adaptable to the user and their environment. Consistency and standardization have helped make it easier for users to understand interfaces in the past, but how will this change if every user has different, personalized interfaces to the same appliance? What is the proper balance between controls on the appliance itself, and controls on the PUC? We do not think that appliances should just become black boxes that can only be controlled from a handheld. There are also cultural issues to consider. How will appliances, homes, and offices change if most control has moved to handheld devices? We plan to study these and other issues over the next several years.

AIWA CX-NMT70 shelf stereo and its remote.

AT&T 1825 telephone/digital answering machine

Panels for the PocketPC to control the stereo and the telephone.


[Bahl 2000] P. Bahl and V. Padmanabhan, "RADAR: An in-building RF-based user location and tracking system," In Proceedings of the IEEE Infocom 2000, Tel-Aviv, Israel, vol. 2, Mar. 2000, pp. 775-784. Adobe Acrobat (pdf)

[Bluetooth 1999] Bluetooth Consortium, Specification of the Bluetooth System: Core, Vol 1B, December 1999, Adobe Acrobat (pdf)

[Brouwer-Janse 1992] Brouwer-Janse, M.D., Bennett, R.W., Endo, T., van Nes, F.L., Strubbe, H.J., and Gentner, D.R. “Interfaces for consumer products: how to camouflage the computer?” in CHI'1992: Human factors in computing systems. 1992. Monterey, CA: pp. 287-290.

[Myers 2001] Myers, B.A., “Using Hand-Held Devices and PCs Together.” Communications of the ACM, November, 2001. 44(11): pp. 34-41. Adobe Acrobat (pdf).

[Nichols 2001] Jeffrey W. Nichols. "Using Handhelds as Controls for Everyday Appliances: A Paper Prototype Study." ACM CHI'2001 Student Posters. Seattle, WA. March 31-April 5, 2001. pp. 443-444. Adobe Acrobat (pdf)

[Philips 2001] Royal Philips Corp., January 2001.

[RemoteCentral 2001],, January 2001.

[Sony 2001],, March 2001


This part of the Pebbles research project is funded by grants from DARPA, NSF, Microsoft and the Pittsburgh Digital Greenhouse, and equipment grants from Symbol Technologies, Palm, Hewlett-Packard, Lucent, IBM and SMART Technologies, Inc.

This research was performed in part in connection with contract number DAAD17-99-C-0061 with the U.S. Army Research Laboratory. The views and conclusions contained in this document are those of the authors and should not be interpreted as presenting the official policies or position, either expressed or implied, of the U.S. Army Research Laboratory or the U.S. Government unless so designated by other authorized documents. Citation of manufacturer's or trade names does not constitute an official endorsement or approval of the use thereof. This research is also funded in part by the National Science Foundation under Grant No. IIS-0117658. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect those of the National Science Foundation.