and Matthew Conway
The class of applications we are examining consists of those that can be driven by two simultaneous analog control signals and possibly an associated on/off switch. All mouse-based interfaces fall into this category. Our approach is called passive tracking, a method that detects body motions by means of sensors worn by the user. We then translate the detected motion into the analog signals that drive the application. As an interim goal, we are currently producing a one-dimensional analog control signal from our users. This allows us to gain experience mapping arbitrary user motions in a simpler realm than the two dimensional case.
This paper presents both quantitative and qualitative results from our first user study, which measures how well children with CP use tailored interfaces to play a real-time video game based on the arcade game Pong. We use a single player game: by moving a paddle, the user attempts to keep a bouncing ball in the field of play. A player's score is a percentage computed from the number of successful blocks divided by the number of opportunities. In addition to our desire for experience with our approach, we ran this user study to demonstrate that with an appropriate input mechanism, our target population could control a real-time task such as a video game. As a benchmark, we compared how our disabled subjects did with the scores of non-disabled subjects. Non-disabled subjects averaged 77% successful returns, and disabled subjects averaged 50%. More than half of our disabled subjects performed better than our worst able-bodied subject. We begin by explaining our approach to motion mapping for one dimensional analog signals. We then describe the user study, including useful information we gained from early pilot studies. After giving a detailed presentation of the results, we discuss several issues that were raised by the performance of this user study.
Proceedings of the ACM SIGCHI Human Factors in Computer Systems Conference, May, 1992, Monterey, CA
One alternative to tracking body motion is to track eye motion. Eye-tracking is not appropriate for our application for several reasons. First, over 50% of cerebral palsied individuals have eye movement disorders [14]. Second, using eye-tracking for the long term goal of CANDY, a speech synthesizer, makes it impossible to maintain eye contact or receive visual stimulation while speaking. Third, many disabled users are poor candidates for eye-tracking because they tend to move their heads.
Gesture recognition has a long history in many contexts, but most research has focused on converting continuous body motion into discrete tokens. Two-dimensional gesture recognition has been used for printed lettering, cursive handwriting, proofreader's symbols, and shorthand notation. In all cases, the approach is to convert the continuous motion of a stylus into a discrete token as input to a language-driven computation or process. Recognition of three-dimensional gestures has also been attempted, but again the main emphasis has been on converting the body motions into discrete symbols that are interpreted as commands to the system [2, 3]. Systems have attempted to recognize static gestures for the deaf alphabet and motions for a subset of American Sign Language. All of these approaches are based on converting three-dimensional signals into a discrete stream of tokens.
Existing work on mapping gesture into continuous control signals is extremely application dependent. For example, advanced military systems exist which map pilot head motion into weapon trajectories. The pilot's faceshield contains targeting crosshairs, and as the pilot's helmet moves with his head, the system computes the angle of his gaze [5]. More detailed tracking is performed in three dimensional drawing or sculpting applications [12], and virtual reality systems, where sensors attached to gloves [4] provide three-dimensional signals that are mapped into motions in synthetic worlds shown on traditional or head-mounted displays. These systems perform mappings from position and orientation information, but the mappings are significantly less complicated than those we propose.
For example, assume that the user had a tracker attached to a wrist, and was told to keep his hand on a horizontal table during the measurement. This effectively constrains his motion to two dimensions. Based on the on-screen display of this raw data, the therapist creates a piecewise linear curve though the data, corresponding to a dominant path of motion made by the user during the control phase. This is done by invoking a heuristic, manually specifying the curve, or a combination of both. All target curves used in this user study were manually specified. Figure 1 shows typical collection of sampled tracker data and the resultant target curves. The first user pivoted his wrist around his elbow, and the second moved his wrist forward and backward.
During the control phase, the user moves the tracker along the target curve and from his position along that curve, we generate an analog signal. One end of the curve indicates 0 percent of this signal and the other end indicates 100 percent. Intermediate positions along the target curve indicate intermediate signal values and the signal generates visual feedback on a CRT. The user is not expected to move the tracker precisely along the curve; we map tracker data to the nearest point on the target curve, as shown in Figure 2. Although in the previous example we limited the user's motions to a table surface, target curves, in general, are three dimensional.
The game was a simple variation on original arcade video game, Pong. A ball bounces off the walls of a square playing field, making a pleasant beeping noise as it contacts each wall. The player defends one of the walls, attempting to block the ball with a paddle before the ball reaches the wall. As shown in Figure 3, each child moves the paddle either vertically, defending the right wall, or horizontally, defending the top wall. On a successful defense, the ball bounces off the paddle and play continues. If the player fails to block the ball with the paddle, it reaches the wall behind the paddle and a less pleasant buzz is sounded. The game pauses for two seconds and then puts a new ball into play at the positions marked with an "X" in Figure 3. In order to avoid a lock step pattern of motion, each time the ball contacts the paddle or a wall, its angle of reflection is a slight variation of its angle of incidence.
One major advantage of using this particular video game as our task is that, although control of the paddle is one dimensional, the player needs to quickly perform a two-dimensional perception and planning task in order to anticipate where to move the paddle. Many people have the incorrect impression that children with CP are all mentally retarded. While the incidence of retardation is higher in the CP population (50%-65%) [14] than in the general population, many children with CP are not retarded. By using this two-dimensional game, we make the point that our target population has the planning and cognitive skills necessary for the eventual two-dimensional version of the Tailor system.
The most difficult question was how to evaluate the effectiveness of our mappings. One design might have been to compare subjects with CP using a traditional input device such as a joystick, with a group of subjects using the Tailor system. This would have been pointless, since we know that our target population cannot use traditional input devices. What we were really interested in measuring is how well our mappings could compensate for disability; how well would a child using our system do in comparison to an able-bodied child of the same age? Therefore, we compared a group of able-bodied children to a group of children with CP who had their input mapped by the Tailor system. One problem with this approach is that if we allowed able-bodied children to use physical devices, such as joysticks, we would also be comparing the performance of the input devices involved. The Polhemus tracker is not as accurate a device as a joystick; more importantly, there is a hardware latency of approximately 85ms [7] which makes the game noticeably harder to play.
In order to keep the tracker lag from dominating our results, we had our able-bodied subjects use the Polhemus and Tailor-mapping software. Of course, the tailored mappings are unnecessary for the able-bodied users; they can adapt easily to any reasonable physical motion. By having both groups use the system, we discovered how big a disadvantage it is to have CP -- how badly are the users disabled when we compensate by allowing them to use their best range of motion. The only remaining issue was that our able-bodied users, having grown up in Nintendo generation, clearly had practice with video games. We counter-balanced this practice effect by not allowing them to play with their dominant hand. We suspect this decision had little effect; none of our able-bodied subjects expressed a strong desire to use their dominant hand. We had two of the able-bodied subjects attach the tracker to their head in order to gain a direct comparison with the one CP subject who used head attachment. Trials were run between February and September of 1991, using children ranging from age six to seventeen, inclusive. Fourteen able-bodied children and eight children with CP participated in the study. We recognize that we are using an uncomfortably small number of subjects in our analysis, but the difficulty in arranging these trials cannot be overstated. Many of our disabled subjects had to travel more than an hour by car to participate in the study. Table 1 shows the breakdown of body sites used for attaching the tracker to various subjects.
The physical setup includes an IBM-compatible 386 personal computer with a color VGA display and either a 14 or 19 inch monitor. The playing field is a square 420 pixels on a side, and the paddle is 95 pixels wide. Our tracker is a Polhemus IsotrakTM [10]. The ball's speed was always one of three fixed values: slow = 33 pixels per second; medium = 64 pixels per second; fast = 178 pixels per second. With each subject, we began at the slowest speed and moved up to faster speeds as the subject became more comfortable. Three of our eight disabled subjects did not play at the highest speed. This was a trade-off between allowing the subjects to practice, and handling their fatigue; due to the relatively short times span the CP children could support vigorous physical activity.
We explained to the children that they were helping us experiment with a new device. We had originally not intended to show a running score on the screen, but our pilot subjects demanded it, and began keeping score themselves by counting. This should not have surprised us, as it is a well known phenomenon that children this age require score keeping mechanisms and invent them when they are not present.
The original game design only allowed the game to be played vertically, defending the right wall. This produced cognitive trouble for the children whose target curves were predominantly horizontal, in much the same way that rotating a mouse 90 degrees makes it almost impossible to use. Other researchers have also found that stimulus and response need to be organized in spatially similar ways [13]. What we found interesting was that the two orientations, horizontal and vertical, were sufficient.
The other things we learned during the pilot studies involved our interaction with the subjects. Having limited or no experience with video games, our subjects had trouble understanding whether they were controlling the motion of the paddle or the ball. We overcame this by having the subjects move the paddle briefly in a training session before the ball appeared. A more subtle problem had to do with retracking. The Tailor system works by using a target curve, and during a session it sometimes becomes necessary to stop play in order to establish a new target curve. This can be caused by many things, including fatigue or substantial motion of the child's body in the chair. With some subjects, when we stopped to establish a new curve they often interpreted this as a failure on their part and needed a healthy dose of reassurance before they could continue. During the trails, we took great pains to avoid the need to re-establish target curves.
Figure 4 shows the results of all trials run at the medium speed, the only speed for which we have data on all our subjects, and the speed upon which we base our overall quantitative result of mean performance of 77% vs. 50%. More important than the mean performance is the large overlap shown between the two groups' performances. Figure 5 shows a similar comparison of all trials run on the high speed; it is less conclusive because three of our eight subjects with CP did not attempt the high speed. Not running at a higher speed seemed to have more to do with fatigue or lack of confidence than performance. The three CP subjects who declined to try the fast speed where not our worst performers at the medium speed; they ranked 4th, 5th, and 7th out of the 8. Therefore, we feel that there is value in ignoring the speed of the trials and lumping them together, the results of which are shown in figure 6. This lumping favors the children with CP, as they include more runs at the slow and medium speed. The full breakdown of trials by speed is given in table 2. In any reasonable interpretation of the data, the children with CP performed much better than we had anticipated at this stage of the project. This is especially true given that most of our subjects had to travel to participate in the study, and several we only obtained because they were traveling to our area for clinic appointments or to have surgery at a local hospital, hardly an optimal time to participate in a user study.
For most of our subjects with CP, this is the first real-time, continuous task they had ever performed.(1) As such, it provides an interesting opportunity to observe their reactions to the pong game.
Most able-bodied subjects would anticipate where the ball was headed and then move the paddle to the correct position and wait for the ball to approach. The children with CP were much more likely to leave the paddle where it had last hit the ball, or at a particular location (near either side wall was common) and then move to block the ball at the last possible moment. We do not know why this particular motor behavior occurred. One possible explanation is that the children with cerebral palsy have difficulty with response programming. More simply, they may be unable to organize and initiate muscular actions to produce a prompt motor response. Alternatively, maintaining the paddle in a "ready" position required more control at an access site [13]. The children with CP typically could not manage this, therefore returned to a consistent resting place.
None of our subjects, either those with CP or able-bodied, had a conscious awareness of the location of the target curve. This is a pleasant observation, because our eventual system will provide open-loop feedback as it is used, constantly determining the shape and location of the target curve as the device is used; the current explicit creation of a target curve is an aberration.
Our most interesting qualitative results focus on the children's reactions to the system. Without exception, they enjoyed the trials. We are encouraged about the approach of passive tracking; although our subjects became fatigued during the trials, we suspect that they continued to perform long after they would have been able to manipulate a physical control, even one built especially for them. In addition, the mapping strategy substantially reduces noise and jitter from the user's motions.
Although our eventual target population is non-vocal children with CP, in this study we included children who could vocalize to some degree so that we could get observations from them. It was from one of the children that we obtained an important insight about passive tracking. Normally, children with CP who attempt a control task "tense up" and hamper their own performance, much like a novice tennis player is often unable to swing properly until he learns to relax. We asked one of our subjects why he was able to control the paddle so well - he replied "I'm not controlling it; it's watching me." His perception was that the paddle was doing the work of following his motions, not that he was doing the work of controlling it. This difference seems to be extremely important in helping the children relax.
Our results were much better than we had anticipated, especially given the adverse conditions our subjects with CP had to endure. In trials with a simple video game based on Pong, the able-bodied subjects succeeded on 77% of their trials, and the subjects with CP succeeded on 50%. Over half the subjects with CP outperformed the worst able-bodied user.
Our future efforts will be in continued enhancements to the mapping techniques used in one dimension, and in producing mappings from motion into a two-dimensional analog control signal.