Summary
Audience interactivity was a primary element of a major planetarium production about cell biology entitled "Journey into the Living Cell." The artist/authors were directly involved with the design of the production from concept to realization. Rob Fisher was the Project, Artistic and Technical Director. Paul Vanouse was Assistant Director responsible for the design and production of the interactive visual portions of the show. Roger Dannenberg directed the interactive audio portions and was responsible for the interactive audio system with the assistance of Jeff Christensen. The following paper provides background about the production and our varied perspectives on the use of the innovative interactive system. In addition, a future production currently pending approval of an NSF grant will be described. This new show about the brain builds upon the experiences gained in the cell project and sheds light on features of audience interactivity that point to some startling conclusions about group behavior.
Background
"Journey Into the Living Cell" was an interdisciplinary collaboration utilizing a planetarium for a new type of science education. "Journey..." was a major collaborative effort involving scientists from Carnegie Mellonís Center for Light Microscope Imaging and Biotechnology, artists from the STUDIO for Creative Inquiry, and educators from Pittsburghís Carnegie Science Center. Similar persons and organizations within Pittsburgh and across the nation were also involved as collaborators, advisors and contributors to the immense set of visual imagery used in the production. The show is a forty minute interactive multimedia presentation on Cell Biology incorporating cutting edge technology that created the Group Immersive Visualization Environment (GIVE).
Educational Goals
The educational aims of the project were to allow large numbers of elementary and secondary school students and non-experts in the general public to experience interactive simulations of scientific phenomena. Specifically, students were enabled to explore the structure and several functions of the living cell. The cognitive goals of the show included teaching the audience biological principles, most importantly: the dynamic, 3D nature and structure of the cell; the cooperative interaction of its component organelles; and the central dogma of DNA and cellular reproduction. The project challenged audiences to correct misconceptions they may have had about the cell, biology, and the practice of science.
The Group Immersive Visualization Environment
A vast challenge for educators and artists alike is the integration of content and form. The planetarium became an analog of the cell, often covered in dense cytoskeletal fibers as if the audience were deep inside the cell. A strategy was developed that allowed for a more participatory learningóinteractivityóutilizing various educational models for its interface. The interactive components of the show functioned like laboratory classes in science curricula, actively reinforcing key concepts following initial presentation of material. Some exercises utilized audience navigation and scaling of materials, in a manner analogous to microscopy. Other interactive lessons required audiences to add materials to the cellular environment as in an experiment to demonstrate ATP synthesis and cellular equilibrium.
Figure 1. The planetarium (circular structure at left) with centrally located video projectors and surrounding loudspeaker systems. The infrared light source and camera system detects light reflected from audience-held paddles. The image is processed by the Cinematrix system, which sends information to the SGI Graphics Processor. This in turn sends data via MIDI to the Interactive Audio System (detailed in Figure 3). In addition, the planetarium operates an independent multimedia system with a vector graphics system (Digistar) and numerous computer-controlled audio, video, and slide projection devices.
The vehicle for these concepts, the Group Immersive Visualization Environment (GIVE,) was a major innovation of the project and the technological point of departure for our current project. GIVE is a synthesis of new visualization and sound tools with existing planetarium media technology. The central component of the GIVE is the "Cinematrix" audience interactive processor and the system of linked technologies which provided graphics and sound which respond to audience involvement (see Figure 1.) The system uses this collective audience response (almost like that of a joystick or mouse), to control graphics projected before the audience. Information is also directed to computers which control full-surround sound, so that sound comes from the same place as the graphics within the domed space.
This technology uses retro-reflective "paddles," an infrared light source and camera linked to a proprietary computer and software system displayed through a Silicon Graphics computer via a high-definition Barco projector. In this technology, the audience simply raises and/or lowers a small paddle. Their "vote" is registered twenty times per second; the aggregate then influences a graphic display. The behavior of the system is such that the audience of 150 begins to feel that they are in control of the graphic which can be as simple as a game of "pong" or as complex as the cell maze used in "Journey...."
Part One. Where Audience Meets Artwork: On the Matter of Interface
By Paul Vanouse
Introduction
The selection or creation of a user interface is one of the most fascinating challenges facing artists working in interactive media today. Unfortunately, it is often understood in utilitarian terms: simply as a method of accessing content and not as a strong carrier of meaning and associations in its own right. Successful artworks must explore this issue and not allow for the standardization and ossification of this defining component of interactive media.
The interface is the juncture of power relations between creator and viewer. Differences between manual effectors and vision-based sensors are examples of two contradictory viewer roles, the former active and controlling and the latter passively surveilled. One speaks of empowerment, the other of manipulation. Obviously, many ambiguous technologies can be cited which fall in-between these two poles. In addition, many input devices, such as buttons, keypads and joysticks carry strong associations with electronic devices which we encounter everyday. Many artists take advantage of the obvious arcade references of coin-ops and joysticks, but many others allow for dangling conceptual loose ends. Whether through harmonious relationships between a userís actions and the artworkís response, or as a point of tension or conflict between facets of the artwork, an interface should be selected that not only works with, but also adds to the artwork. If the form of the interaction does not add to the meaning of the work, perhaps the work shouldnít be interactive.
In "Journey Into the Living Cell," we used a real-time, mass-audience polling device known as Cinematrix as our point of interface with our audiences. We incorporated the Cinematrix hardware into a total gestalt known as the Group Immersive Visualization Environment. In conceptualizing and directing interactive scenario development for the project, my objectives were to balance pedagogical content with aesthetics, while also trying to explore the unique meanings and novel characteristics of GIVE.
These objectives are similar to those of my independent artwork. In this work, much importance is placed on the interface which I feel can never be transparent. The interface carries important meanings which are absolutely integral to the understanding of the artwork. This exploration of user interface begins with two examples from my independent work, then discusses two interactive scenarios from Journey Into the Living Cell, and lastly addresses the future of the collaborative team and the GIVE.
Independent Work
"Items 1-2,000: A Corpus of Knowledge on the Rationalized Subject," 1996, is an interactive installation combining sculptural, pictorial and performative elements with custom software and electronic circuitry. It collapses Western medicineís fracturization of the body with industrial itemization techniques into a strange rationalization apparatus. A human body is half submerged in a block of wax, in a manner reminiscent of how biological specimens are fixed in a "microtome" (a machine that cuts specimens into thin slices.) A sheet of glass rests inches above the figure in a manner analogous to a cover slide used atop cross-sectional slices in microscopy. This glass is affixed with barcodes which correspond to internal organ locations of the figure underneath.
Participants interact with the work as anatomy students would a cadaver. They use a stainless steel barcode scanner much like a scalpelóslicing horizontally across the glass to reveal the hidden target organ on monitors resting on laboratory shelving. The more familiar use of barcodes and scanning procedures however are not lost, and this surgical role blurs with that of cashierócommodifying and extracting value through the denial of the body as whole (rather a rational composite of itemized parts.) Every third scan the participant makes accesses a video recollection from my own experiences in the anatomy morgue. These recollections are somewhat poetic and address the phenomena of de-humanization of the corpse as it is de-constructed and re-configured through dissection. The artwork seeks to contextualize work in anatomical imaging, using the Visible Human project as an example, with the social issues of American medicine, and my own observations as a pre-med student in the 1980ís.
The barcode and optical scanner interface used in the project had both associative meaning and technical utility within the work. Such a device was intended to superimpose economic connotations upon the body of the subject (a death-row inmate was used in the Visible Human Project) and the broader field of medical imaging. Users felt an uncomfortable tension between the modelís prostate body and this instrument of consumer culture. In the Visible Human project, the body was transformed into pure data by the slicing/photographing/digitizing processes and the barcode interface of the artwork provided a method to both itemize and quantify that information.
Additionally, as the work neared completion and the barcode interface moved into its final considerations, more subtle choices were made about which specific scanner to use. High-quality commercial scanning-guns utilize several lasers so that users may simply aim at the target barcode to read its data, while cheaper, one laser pen-models require the user to actually drag the unit over the barcodes at an even speed. The more sophisticated guns however carried different meanings than I necessarily sought in the work, for instance any time we aim at a target we reference warfare, handguns, etc. The pen on the other hand required users to actually glide across the body in a manner much like using a scalpel. When users occasionally had difficulty getting the pen to register their scan, they were un-flustered, perhaps because the surgical role was understood to require both patience and skill.
"The Consensual Fantasy Engine," 1995, is a computer program capable of creating a cinematic drama based on audience preferences. The engine asks the audience questions to which they answer by applauding for their preferred choice. The engine responds to the applause level by creating a customized scenario for them in seconds and projects it onto the cinema screen. Every 5 minutes the audience is invited to respond to questions of a sociological nature and influence the plot. The point of departure for the narrative is the O. J. Simpson chase which the audience can transform into a Bonnie and Clyde style road adventure, an intriguing, Film Noir search for the real villain, a suspenseful trial, or any of millions of intricate variations.
The televised O. J. Simpson chase is important because of a distinct relationship that it shows between the broadcast and society. Societyís presence as viewers had an impact on both the actions of the police following, and also possibly on OJís own actions during the pursuit. Our presence as viewers has changed the courtís jury selections and even the selection procedure. Most importantly however, our responses to (or public opinion of) the constant barrage of informationótruthful, real, circumstantial or fabricatedówill set up lasting metaphors and prejudices which will affect our understanding of future world events. "The Consensual Fantasy Engine" explores how both the media and the public have a substantial stake in the creation of such metaphors and meanings.
"The Consensual Fantasy Engine," as a mass-audience interactive artwork, has many formal similarities to "Journey Into the Living Cell," as well as many differences in content and intent. The software applause meter user interface was created very early in the workís conception in strong preference to any type of distributed push-button system. The primary reason for such an interface was the mass spectacle that the work addressed. The response to such spectacle has always been applause and shouts of support, not unlike that delivered along the roadside during OJís flight from the police. Secondly, it related to O. J. Simpsonís history as an athletic superstar and referenced memories of sports fans shouting support for their heroes.
A somewhat unforeseen attribute of the system was that it allowed for analog input from the audience. They could clap a little or a lot, and they could further augment their vote by banging on a table or whistling. Thus opinion was scaled by enthusiasmólike a democracy in which the loudest voices had the most input. The real-time metering of response further incensed audiences to vocalize their opinions since they could note how their own participation caused the meter to register higher and thus increase the average noise-level used by the system to decide the winning response.
Journey Into The Living Cell
"Journey Into the Living Cell" certainly posed many challenges related to merging form with content. I feel that several realizations which occurred to us early in the development stage led to our ultimate success. Firstly, we decided that these interactions should work like laboratory classes in science curriculaóreinforcing fundamental principles through direct involvement. Secondly, we realized that the interactive scenarios must co-evolve with the storylineónot written before and inserted as an afterthought, or envisioned afterwards, like illustrations, to support the text. Thus the entire show was conceived around balancing content with its proper form of interaction and balancing pedagogy with the excitement of interactivity. We wanted to make sure that the interactions were never gratuitous, nor were they purely illustrative or predictable.
During the Cell Membrane interaction, audience members hold up their reflective wands so that "ions" mapped to their seats enter through the membrane into a simulated cell, and hold their paddles down to bring them back out. The cell expands, somewhat balloon-like, when too many ions enter the cell and it buckles and folds inwards, deflated, when too many ions leave. A slider bar at the bottom of the screen further assists the audience in understanding the cellís state. The audience attempts to achieve a state of equilibrium to create a healthy cell, into which they will soon be traveling.
The scenario highlights an interesting phenomenon we have observed (in all 6 of the interactions) of spontaneous cooperation. Most audiences seem to easily consolidate their efforts to achieve correct concentrations. Whereas most of the scenarios in the show require a quorum of responses (similar to voting) to effect the desired change, this interaction requires them to achieve nearly perfect balance of up or down paddle response. This is a fundamental attribute of the real-time Cinematrix system that found a perfect match in the concept of cellular equilibrium. Such a result could not be achieved within a vote-and-wait system.
Additionally, the exercise took advantage of the systemís capability to individuate audience members. Each audience member can actually see the simulation respond to his or her individual action since each audience member controls the state of a "personal" ion. Thus, unlike many examples of mass polling, the user gets instantaneous feedback from personal actions, while also realizing a role in the larger task.
The ATP Production in Mitochondria interaction had dual objectives: To reinforce understanding of ATP synthesis within the mitochondria and to excite the audience for the grand finale of the show. The audience is required to expend their own ATP (by rapidly moving paddles up and down) to run sugar and oxygen pumps in the mitochondria. The actions of such pumps lead to the production of ATPóthe energy currency of the cell. Audiences see the sugar and oxygen molecules enter an immense mitochondrion shown at center screen (see Figure 2) and the ATP molecules shoot out like fireworks. The faster they move their paddles the more ATP is produced and the faster an animated figure runs around the mitochondrion track.
While most of the showís interactions attempted to maximize strengths of the Cinematrix system that were largely in keeping with the systemís design, the ATP interaction exploited a simple, yet unexplored possibility. Rather than be concerned with an audience memberís response of up or down, we wrote software that kept track of the rate at which members changed their paddle positions from up to down. This established a gradated method of judging enthusiasm and excitement, not just a binary choice.
This curious method of interaction was metaphorically linked to the mitochondriaís role of pumping sugar and oxygen in the production of ATP. The audience was required to expend energy, or ATP, by rapidly pumping their paddles to create the bodyís energy currency. (If this seems abstract, wave your arm up and down vigorously for about one minute and you too will become keenly aware that sugar, oxygen, and ATP are not just biological abstractions.) Whereas success in other interactives within the show was based on achieving balance or successfully navigating the cell, success in this exercise was based on enthusiasm. In addition to the linkage between the biological concept and the technological means for this lesson in energy production, there was a very pragmatic reason for such an exerciseóto shake some life into youngsters who were perhaps drifting off near the showís ending.
Figure 2. Scene from the ATP Production interactive segment. Audience members wave their paddles to pump oxygen and sugar into this mitochondrion, causing the man to run around. (Mitochondrion created by Anoop Bhattacharjya from data set courtesy of Wadsworth Center of the New York State Department of Health.)
Conclusion/Coming Attractions
I have found that the most interesting interactive works address the notion of decision-making, choice, behavior and the analysis of these actions. Nearly all members of the Journey team began to recognize the strong psycho/sociological connection inherent to the GIVE and the interactive genre. Team members were always noting what audiences found exciting, what they were learning from, how their roles needed to be explained, how to motivate them to participate and how they communicated among themselves.
Our new project, tentatively titled "Tracking the Human Brain," realizes the strong psychological implications of interactivity and will attempt to make the form the content, and the content the form. For example, in explaining the phenomena of neural networks, the individual audience members will be allowed to act as neurons in a network which is in fact the audience. To do so, the system detects audience memberís actions and uses their actions to pass information along to other audience members. Thus the system acts like a chemical (neuro) transmitter passing signals between neurons. This is just one of many possible scenarios planned, in which we hope that tool and content, message and medium are reciprocal in adding the most interesting associations and meanings to make a ground-breaking, communicative, and educational artwork.
Part Two. Audience Interactive 3D Audio and Sound Design
By Roger Dannenberg and Jeff Christensen
Sound is an essential component of any theatrical or multimedia experience. As part of the team that created "Journey Into the Living Cell," we had the opportunity to coordinate sound effects and music with real-time animation in a planetarium with multiple speaker systems. We developed a custom sound spatialization system and integrated it with other elements of the production.
This part describes design considerations and the technical aspects of the spatialization system. In addition to the technical aspects, we will discuss aspects of the sound design that led to the actual sounds used. In the process of integrating this work with that of many others, a number of modifications and extensions were made to the original plans. We want to describe our experience in the context of developing a large interactive multimedia presentation.
A Review of Spatial Audio Technology
As computer graphics technology evolved to provide interactive, real-time, 3D, animated, and even stereographic images, many researchers turned their attention to the medium of sound to study how the audio channel might enhance experiences with interactive graphics. Although interactive sound has been a topic of study in the field of computer music for many years, a new stream of work has been motivated by interactive graphics and virtual reality applications. This stream has largely focused on individuals listening through headphones, and some interesting technology has developed. The basic approach relies on the fact that sound waves are altered by our heads and ears according to the direction of the sound source. Digital audio processing can synthetically reproduce these directional cues and provide very convincing "3D sound." Normally, headphones must be used to provide a separate audio channel to each ear and to eliminate room effects. [Begault 1994]
Unfortunately, because of headphones, the effect is spoiled when the listener turns his or her head: normally when we turn to face a natural sound source, our head has a new angular relationship to the source, and the sounds that we hear are altered accordingly. With headphones, there is no change in the sound when we turn our heads. We can simulate the desired effect, but ordinarily this requires tracking the listenerís head position and constantly compensating for changes in angle.
Thus, what has become common practice in 3D sound systems relies upon head tracking to sense the head position and headphones to deliver sounds to individual listeners. Since audiences do not expect to wear headphones and audience members do not move their heads in synchrony, this approach does not scale well to groups.
There are also "3D" systems intended for use with loudspeakers, but these generally rely upon some kind of wave cancellation effects, which in turn rely upon careful speaker and listener placement. (Kendall 1992) In general, only a few audience members would experience the full effect of these systems for 3D localization, although the effect is sometimes quite interesting in other ways.
Finally, there are various "surround sound" systems. (Dressler 1996) These use multiple speakers to give a sense of direction, and the technology is mainly used to combine the information for many speakers into only a few channels for recording purposes. These systems do not address the general question of producing or controlling the original multi-channel program, but the idea of using multiple speakers for sound spatialization is the basis for our approach. For further information on the general topic, an excellent guide to on-line material and examples is: http://www.dform.com/inquiry/spataudio.html.
Our Sound Spatialization System
The GIVE sound system implements old ideas described by Chowning (1971) using new technology. Others have created similar systems (Bossi 1990, Perez, et al. 1996), and we do not make any claims of innovation except possibly that we have created a low-cost system that is controlled by audience members.
In Chowningís system, several cues are used for the spatial location of a sound. First, sounds are panned across multiple speakers according to the angle of the (imaginary) sound source. Second, the overall loudness of the sound falls off with distance. Third, the ratio of reverberation to direct sound increases with distance. Finally, simulated Doppler shift is applied to moving sounds.
We created a system that implements all of these cues in real-time using audio from a sampling synthesizer. Up to 4 sources can be independently spatialized and controlled via MIDI. We looked at various implementation strategies and settled on using MIDI-controlled analog mixers. Our system has a total of 4 "dry" audio inputs. Each of these is run through a separate reverberator. The eight resulting signals (4 dry and 4 wet) are routed through a full 8-by-6 analog crossbar mixer to yield 6 output channels. Four outputs are used for front-left and -right, and rear-left and -right speakers, a fifth channel is directed to an overhead speaker, and the sixth channel is directed to a speaker array located on the floor under the seats. The 8-by-6 crossbar means that each of the 8 input signals can be routed in variable amounts to each of the 6 outputs, allowing completely arbitrary panning and level control.
An all-digital implementation would require a total of 10 input and output channels combined with the computation to reverberate, control, and mix 48 patch points. This approach seemed prohibitive or at least risky. Instead, we chose a hybrid approach, shown in Figure 3. Low-cost MIDI-controlled line mixers intended for use in mixer automation (the CM-AUTOmation model MX-16) offer a mode in which a single rack space unit provides two 8-to-1 mixers. Thus, 3 MX-16 units and 2 dual reverb units provide the required capacity, and it was only necessary to make special audio patch cables to configure an 8-by-6 mixer.
Even though there are several papers in the literature about similar systems, interesting problems still had to be solved. One was how to extend Chowningís 2D configuration to 3D using an overhead speaker. After considering various formulations, we settled on a simple one: first, ignore the vertical component and compute panning coefficients for the four speakers in the horizontal plane. Then, consider only the vertical component to pan between the horizontal plane and the vertical speaker. Combining these two calculations gives weights for five speakers. (The sixth channel for under-the-seat speakers was only used for special effects.)
Figure 3. Implementation of the Interactive Audio System. An 8-by-6 analog mixing matrix is used to route and reverberate sound from 4 sources to 6 loudspeaker systems.
Another issue is how to pan smoothly between speakers. Chowning describes the "hole in the middle" effect where the sound seems to "drop out" between two speakers if panning is done in a linear fashion. He suggests a particular formula to boost the overall power as the sound is panned between two speakers to compensate for this loss. (As an aside, there is a physical explanation for this effect: if you direct half of the amplitude to each of two speakers, the total average power will be cut in half relative to full amplitude in one speaker.) Rather than adopt Chowningís exact formula, we made the "center boost" a variable that we could adjust in the theater. It turned out that this was a fairly sensitive parameter.
We did not find any discussion of locating sounds within the auditorium, and the standard "1/radiusn" formulas "blow up" at a radius of zero. In our system, when a sound is inside the speaker radius, signal is added to all speakers and the level increases linearly as the location approaches the center of the space. We found that this gives a nice, even visceral, effect when objects zoom from a distance through the center of the audience space, especially when combined with Doppler shift.
Another interesting phenomenon is the tendency of sound to perceptually "jump" from one speaker to the next during a smooth pan. To compensate for this, it is a good idea to pan rapidly to the neighborhood of 50/50 where the jump occurs, and then pan very slowly across the 50/50 point to minimize the jump effect, and finally pan quickly to target speaker. A "center dwell" parameter allows us to vary from a smooth pan to a one that spends most of the time near the 50/50 point. Again, this parameter can be adjusted by listening in the theater.
Figure 4 illustrates the output of the system to four different speaker locations in the horizontal plane given an input that is traveling in a circle around the center of the space. Notice that the "center dwell" effect has flattened the curves between peaks and sharpened the peaks (where the source is aligned with a single speaker). Also the "center boost" raises the levels slightly between peaks.
In addition to the non-linearities that we intentionally add to the panning control, the MX-16 mixers have a non-linear (and non-logarithmic) response to the MIDI control values. This is probably appropriate for a fader automation system, but we needed the mixer to actually deliver a good approximation to our mathematically calculated attenuation factors. Since the manufacturer was unable to provide detailed information, we resorted to measuring attenuation with a digital voltmeter. The measurements were translated into a table of (software) values that converts desired attenuation into the appropriate MIDI control value.
Figure 4. Output levels to 4 channels in the horizontal plane from a source moving in a circle around the space. (Time is represented on the horizontal axis.) Note the intentional non-linearities.
To adjust parameters, a program moves a sound in a virtual circle, causing smooth continuous panning around the four horizontal speakers. Parameters are adjusted to give a smooth perceptual motion without any pulsing (the "center boost" adjustment) or jumping (the "center dwell" adjustment).
Doppler shift is implemented using MIDI pitch bend control on the source (a sampling synthesizer). In our application, we were not trying to mimic a particular size of space or speed of motion, so we made the "speed of sound" a parameter. Lowering the "speed of sound" increases the Doppler effect. In general, we avoided any attempt to calibrate sound effects using physics. It seems that in an environment like this, perception is the only guide, so we adjusted parameters freely to achieve interesting and convincing effects.
Using the System
Our spatialization system consists of a Macintosh running a program written using the CMU MIDI Toolkit (Dannenberg 1993), but this is only part of the complete system shown in Figure 1. Originally, we planned to provide a general interface that would allow the animation software to specify the location and velocity in 3D for each of 4 sound sources. These could all be transmitted using various MIDI control change commands. What we soon discovered was that it made much more sense to develop specific protocols for each interactive segment of the show.
For example, there is one interaction where the audience waves paddles, causing a man to run around in circles (described more fully in Part 1 above and depicted in Figure 2). We developed a very simple protocol in which the animators only need to send out a start command, the angular position of the runner and the speed of the runner. Using this, our local software sends commands to generate heavy breathing and footsteps and moves them in a circle around the audience. With this approach, the details of the audio could be developed and tested independently from the animations, and the additions to the already complex animation code were minimized.
In general, the more self-contained we made the audio and its control, the more we, as sound designers and implementors, could test and enhance the sound without coordinating with the animation team. When schedules slipped (as they always do) and the deadline was looming, there was little time for further tuning, adjusting, and reworking. Although things worked out well, we should have anticipated this outcome from the beginning.
Sound Design for Audience Interaction
One of the challenges of the show was to come up with not just a few effects but a total sound environment for a complicated and fast-moving program. In terms of the implementation of the sound design, the show can be broken into two main parts: linear and non-linear sections. The linear section has a narrated script and soundtrack (by John Gore) that runs throughout. The bulk of the narration is handled by an adult "tour guide" figure, with two youngsters interjecting questions and comments. The tone of voice is conversational, and the characters have definite personalities. To create the impression of listening in on this conversation as it takes place, the voices are located in distinct places: the main narrator is centered in the overhead speaker channel and the two children flank her on either side. The comments from the young people often correlate directly with images on their particular side of the screen, which amplifies their physical spacing.
The non-linear interactive sections are interspersed throughout the show, so there are breaks built into the script and soundtrack. This void must be filled with a sound design environment that enhances the content and engages the audience. These environments vary in complexity and usage depending on the context of the visual content.
Some segments primarily call for feedback sounds that relate to direct manipulation of objects on the screen. This feedback reinforces the interaction techniques of the paddles, which is an important part of the early show segments. For example, as the audience moves an on-screen slider up and down, it is important to create a sound that mimics the movement. This creates an interaction that feels natural and intuitive, and gives the members of the audience a sense of empowerment and individual control. Also, as the visual imagery grows in complexity later in the show, aural feedback of direct actions gives immediate reinforcement on group activity without forcing collective visual focus on one area.
Designing sound for a show on molecular biology is not an opportunity for literal mimicking of sound attributes, as it is doubtful anyone could rightly contest the actual sounds of a ribosome versus that of the golgi bodies. During the development of the script, it was decided that metaphor was the appropriate vehicle for both the written script and the sound design. A city was chosen as the illustrative object, with each cellular entity playing a different role, such as manufacturing, energy creation, communications hub and distribution center. This metaphor presents lots of rich sound possibilities and the relative functional connections can easily be made. As the imagery in the show is primarily scientific in nature, the sound has to be a constant reminder of our metaphor to be sure the audience retains relative organelle functionality.
These metaphorical sounds were used to illustrate the functional relationships between cellular parts in both the linear and non-linear sections of the show. Layered sound environments were created for each of the six highlighted cellular elements, which are included on the 8-channel tape in the appropriate spatial locations when featured in the linear segments. We also created long non-repeating beds on two compact discs for the transitions to and from the interactive sections. For the non-linear sections, these layers are played on the fly using a generated MIDI stream through a Kurzweil K2000 sampler. In order to eliminate audible loop points, each organelle has 1 to 3 continuously looping sound layers, each offset and of different lengths, on top of which anywhere from 3 to 20 more individual non-looping sounds are played. Some organelles play back a specific MIDI stream to recreate a precomposed pattern, and others use algorithmic music generation techniques to create pseudo-random sequences that never repeat during the show.
Fortunately, the K2000 sampler can be used to pre-mix a complex texture representing an organelle or any other sound source. In this way, even though our spatialization system has only 4 inputs, each input already represents a rich composition of sound.
With the limited RAM capacity of the sampler and a palette of just under 100 sounds, 16-bit 32 kHz sampled sound files were used as a compromise between file size and sound quality. All sounds are monophonic, as they are placed in space with our spatialization system. To preserve sound integrity, all sounds went through analog to digital conversion only onceóeither recorded digitally locally or digitally extracted from a sound effects library disc. All pre-layering, editing, looping and effects processing were accomplished with Passportís Alchemy and Digidesign ProTools on a Mac, and when completed were dumped directly to the sampler via SCSI.
As an example of how these sounds are used, in some of the interactive segments the audience pilots a large LDL molecule through a cell to certain destination organelles. Sound localization is used as a navigational aid, as each organelle has a distinct aural flavor. Sound spatialization is based on the distance and direction from the LDL molecule to all surrounding organelles. Since localization is stronger from front-to-back than up-and-down due to speaker placement, the plane of reference of the visual image is rotated so that "up" in the visuals corresponds to "back" in the audio. (This is not so strange to the audience: continuing "up" on the planetarium dome leads to the "back.") Left and right visuals correspond to left and right audio as would be expected. As the audience sees the cell nucleus to the right, they will hear the metaphorical "communications hub" sound environment approaching from the right. If they collide into the nucleus on the screen, they will hear a collision sound on their right, accompanied by a louder (closer) nucleus environment. This real-time mapping creates an intuitive orientation and navigation tool to aid the audience interaction.
Evaluation
Overall, we feel that our results are effective, and the sound certainly makes an essential contribution to the audience experience. The implementation approach met our needs, but it would have been much more interesting to spatialize all sounds rather than the limited set we could generate via the K2000 synthesizer. In the future, faster computers will make this possible by taking over all storage, processing, and playback functions.
The sound design resulted in a rich palette of sounds that seems to help the audience relate to the educational content. The continuous but subtle changes of location and reverberation of up to four independent sound streams further enriches the experience. In "Journey ...," we conservatively limited spatialized sound to the interactive segments. We look forward to future projects where spatialized sound can be used continuously and seemlessly for all audio facets of the production and where all these facets will be interactive in some way.
Part Three. Observations About Audience Interactivity in the Cell Project and Beyond
By Rob Fisher
Introduction
Audience and critical acclaim has followed the past year of daily presentations of "Journey...." But like most artists, one gets dissatisfied and perhaps hyper-critical of oneís first efforts and wants to push on and evolve the art form. Such is the case with audience interactivity. As a project team, we felt we just uncovered the beginnings of very powerful possibilities. Urged on by positive response and encouragement from the National Science Foundation, the team developed a proposal for a new production tentatively titled "Tracking the Human Brain". In this new production, several very exciting partnerships were formed between the STUDIO for Creative Inquiry, the Center for the Neural Basis of Cognition and the Pittsburgh Supercomputer Center. But the most exciting aspect of the new production about the Brain is the opportunity to further explore and nurture what we discovered about audience interactivity in the Cell project. A valuable part of this case study is the evolutionary process leading to the next generation production, an evolution based in part on objective as well as personal critiques of the cell project from those who worked closely on it.
Positive Response to the Cell Show
The cell project was very well received in the scientific, education and art communities and certainly within the general public. It is estimated that as of October 1996 over 20,000 persons have seen the show in its original venue, the Carnegie Science Center. The program will continue to be presented both to schools and to the public during 1997 as well. Additionally, over 35 copies of the show have already been distributed to other planetariums worldwide. Ten thousand resource guides were printed for school groups attending the show. "Journey Into the Living Cell" has been covered by numerous newspapers, journals and other media including The Wall Street Journal, Wired magazine, and CNN. Additionally, the work has been presented and discussed in various national and international conferences such as Association of Science and Technology Centers, 96; the International Symposium for Electronic Art, 95 & 96; the annual meeting of the Society for Literature and the Sciences, 96; and the Fifth Biennial Symposium for Arts and Technology at Connecticut College, 1995.
NSF Evaluatorís Comments and Remediation
An NSF evaluator was hired to formally evaluate the Cell show. He was very positive about the show overall, stating in his "critical appraisal report" that "One would have to ëdeclareí the cell show a success on the basis of its initial positive impact on the audience at the affective level and its coverage in a visually rich environment of a large amount of factual information relating to the workings of the living cell." However, there are several criticisms from his evaluation that are very pertinent to the future Brain project. They are also concerns which have motivated further refinement of our production model. They are certainly common criticisms regarding the development of active learning tools.
Our evaluator felt that audiences needed to be introduced to the interactive system more fully prior to the show. Otherwise what was being done takes time from why it is being done in the show. In other words, unless the audience has an understanding of the interactive system beforehand, the learning of the system appears to detract from the conceptual learning to be achieved through the interactions. This problem was subsequently addressed by increasing the training of planetarium presenters to include two preliminary interactive sequences, the main focus of which is getting acclimated to the new paddle interface.
The first sequence simply lets the audience see the camera image of themselves and lets them find their seat by holding up their paddles row by row. There is no show-specific content to this interaction. In this way, audience members better understand how their paddle raising is tabulated by the computer.
The second sequence is a very simple exercise in which the entire audience controls a slider on a long slider bar. The presenters begin this sequence by having the audience position the slider at various points on the bar to get comfortable with the notion that to do so requirs a cooperative effort: half the audience holding up paddles and half keeping them down. This exercise has slight pedagogical content (it teaches about the dimensions of a cell), but the main focus is getting acclimated to the new paddle interface.
The evaluator noted that segues between interactives and the main show needed to be carried out in a smoother manner. Initially, the planetarium dome went dark and the soundtrack ended before the interactive exercises began. Greater smoothness helps the audience understand the relationship between the interaction and the preceding linear story. It was also felt that elements from the soundtrack should be carried into the interactions. An audio CD was produced, which was programmed into the show at the moment of transition, including student explanations and soundtrack elements, which solves this dilemma. In addition to fixing the gap, this also allows the young voices to actually explain what the audience should do. This relieves the presentersówho are generally less informed of biology than the astronomic content of most planetarium showsófrom the burden of responsibility for content.
Audience Reaction to Interactivity in the Cell Show
Reactions of the audience to the interactivity system in the Cell show ranged from enthusiasm to nonchalance, to disbelief. It is the latter reaction that was for us the most intriguing. In this group the observer simply did not believe that there was any connection between the actions on the screen and his individual paddle. Some went so far as to imply that the interactivity was non-existent and that it just appeared to have a correspondence. An audience, in these peopleís minds, could never arrive at a clear and precise consensus in such a short time. (For example, the audience can readily position a slider along a comparative scale of sizes or find its way through a complex maze).
This is a good lesson for our design team as well as any other developer of interactive devices. Namely, users have to believe that they are recognized by the system and must be able to observe their personal impact. Given this challenge, in the context of our new project the design of the interactive interface will be subject to extensive research and refinement. The nature and effectiveness of the feedback that the audience receives will be thoroughly examined to maximize interactivity, motivation, and capability. We want the audience to succeed. The basis for the Brain project is that the audience can itself be used as an educational medium revealing to themselves, through their own behavior, many if not most of the major features of the brainís modus operandi.
Audience Interactivity in the Brain Project
The most innovative aspect of the new production will be the use of the interactive audience technique, "Cinematrix," introduced successfully in the "Journey into the Living Cell." It is our intention to expand the innovative, interactive system that has proven itself to be both workable and highly entertaining. Both the developers of Cinematrix and members of our team have intuited a potential relationship between the way in which the brain neurologically processes information and the way in which the individual audience members, using the system, unite to make a series of decisions. The possibility of using interactivity to provide an "x-ray" of the audience to itself and concurrently present complex information in an entertaining fashion should advance the field of entertainment and interactive technology significantly.
Evolution Of Delivery System From G.I.V.E. To G.I.B.E.
In the evolutionary design of delivery systems for science education, a guiding principle we have followed is the appropriate and subject-specific application of technology. In this new project about the Brain, which builds on the results and success of the prior NSF supported "Journey into the Living Cell" planetarium show, the change of subject matter from cell biology to the Brain and its behavior has led us to re-evaluate the previous delivery approach entitled GIVE (Group Immersive Visualization Environment). In the evaluation process we have isolated the key elements of GIVE that apply to this next generation project and proposed a highly refined multimedia metaphoric environment we call GIBE (Group Interactive Brain Emulator). The challenge is to align the presentation system with the subject matter and information so that the application of new A-V technologies is not gratuitous. Thus, while it made sense to immerse the audience inside a cell in order to suggest the liquid environment and the cellís architecture, the approach we are considering for the Brain focuses much more on task-oriented interactivity coupled with an enhanced stereo projection system that places the emphasis on the decision-making process and seeing and hearing. Special emphasis is given to this innovative scenario for presenting the subject matter. In an age of simulators and other virtual reality type entertainments, it is our belief that such an approach will capture the attention of a younger audience while the subject itself should find appeal in every age group.
Group Interactive Brain Emulator
The Group Interactive Brain Emulator transforms the planetarium into a virtual decision-making robot (a term coined by Herb Simon) whose task is to manipulate information in the form of images and sound and whose controls are in the hands of audience members who represent individual neurons and as a group become a metaphor of the brain.. The audience is both pilot and passenger, enabled by the technology to guide itself as a group through a lesson on the Brain. The environment is like a cockpit in which the audience is seated in a clearly segmented arrangement based on the brainís architecture with right and left hemispheres and a frontal lobe. The projection surface/audio space acts as an interface between the group and information about the subject matter. It offers navigational cues as well as a virtual stereo three-dimensional window in which images appear and events happen. There is no narrator. Voices, interactively responding to the audience actions, provide an "inner voice" that leads them or corrects them. It cajoles, admonishes, encourages, becomes irritated. The environment takes on a persona, a sense of presence, as if it is alive.
Rationale
The rationale for this shift in media emphasis is several-fold. Our interest in the Brain project is to demonstrate the manner in which the brain processes information. The emphasis on process finds its precedent in the interactive tasks developed for the cell project. In these tasks the audience was invited to use the interactive system as a tool to measure scale, as a navigation device to guide a molecule through a cell maze, as an ion exchange mechanism to expand or contract a cell membrane and as a pump to provide a cell with ATP and cause a running figure to accelerate or decelerate. These portions of the cell show were engaging, popular and novel as we believe this to be among the first times that group interactivity was coupled with science learning.
Audience As Medium
Observation of audience behavior during Cell show performances has suggested an approach for the Brain project that would utilize group behavior patterns, emergent behavior (leadership), and decision-making by a large group in which there is no traditional face-to-face communication.
Enhancement Of Audio/Visual Feedback
Enhancement of audio/visual feedback is another area of research that derives from observation of the audience during the Cell project. Navigational "directional signals" introduced during the cell maze assist the audience in recognizing the overall behavior of their input. The audience is divided in half with the left side of the audience controlling the up/down movement of the molecule and the right side controlling the left/right movement. In the upper corners of the screen, arrows point left and right or up and down as an aid to the audience. In the Brain project, the metaphoric architecture represented by the seating layout will when appropriate be projected on the screen, revealing to the audience patterns of their own behavior as it occurs. Points of light that are a picture of the reflective paddles being held up at any time will be displayed in a shape that corresponds to the seating layout so that there is a clear understanding, a one-to-one correspondence between audience member and visual image.
Use Of Audience Interactivity To Allow Experienced Based Exposure To Key Concepts
Audience members can be divided into groups corresponding to brain regions, and persons in each region can be asked to respond to particular sorts of information (visual, auditory, etc.). This will convey the idea of specialization of function. Within regions, individuals can be assigned more specific functions, e.g. to detect stimuli in particular locations, thus illustrating the topographic mapping of space within various brain regions. Combining input from multiple parts of the audience will allow illustration of cooperative action of the brain. Reassignment of roles of neighboring parts of the audience when one part is temporarily shut out will be used to illustrate plasticity and reorganization after brain damage.
Enhancement Of Pedagogical Techniques
The system described above fundamentally enhances traditional pedagogical strategies. Firstly, it allows for an active participation by the audience; they can actually manipulate objects of study. We liken this to the value of laboratory experiments in scientific education. Secondly, this interaction is special in that rather than encouraging solitary enjoyment, as do computer games, it promotes a social interaction. Audience members talk back and forth and will often coach one another. And lastly, and especially relevant to our focus on the brain, it will allow audience members to experience metaphors of neural response and perceptual thresholds as they see the responses of their neighbors projected on the screen as navigational cues. For example, we envision making each seat analogous to nerve cells, so by holding up their paddles at correct moments (as nerve cells would transmit charge in the presence of the proper chemical,) they could possibly assist in bringing an image into focus or a sound to the audienceís threshold of perception.
Intelligent A-V System
Building upon the interactive system used in the "Journey into the Living Cell" project, an innovative aspect of the present project will be its adaptive nature. We propose to develop an "intelligent" system in which the difficulty or constraints on any given audience task is adjusted continually throughout the event. In the Cell show the live operator can adjust the difficulty of the maze sequence on observing the skill level of the audience. In the Brain, we plan to create an intelligent A-V system since the technology can provide a continuing portrait of audience behavior. Each vote cast can be monitored and recorded, both digitally and visually, presenting a data picture over time. The next generation system will evaluate the skill of the audience in realtime and alter contents, rate of presentation, and other factors accordingly. This will be done through the use of neural network software that will provide a realtime updating of system parameters to create "tasks" that are a function of the information provided by the audience. These "tasks" will be geared to the audienceís skill and knowledge levels as revealed by their ongoing responses. This type of "biofeedback" system will serve to both illustrate how the brain functions and to "build" the presentation. Such adaptive learning systems are becoming more popular and feasible as exemplified by their use by the Educational Testing Service to administer the Graduate Record Exams at computerized centers across the country.
Planetarium Dome As Metaphor
Use of the planetarium surface with its domed shape, is an apt metaphor for the brain with different brain areas assigned to different parts of the planetarium dome. "Journey Into the Living Cell" laid out the parts of the cell on the dome surface; the brain lends itself even more to this because its surface maps nicely onto the shape of the dome. Patterns of brain activation can then be displayed illustrating the involvement of each part in different functions, such as perception, memory and language. An aspect of our research will be to see if this can be accomplished in realtime. The highly refined planetarium computer operating system, network of cable and connections, software and multiplicity of audio and visual hardware each have their analog in the human system.
Development Of A Studio, Lab And Classroom
During the evolution of the cell project, a one semester advanced class in Audience Interactivity was implemented at CMU to train, write software, debug the interactive system, learn to install and operate the system so that its operation could be learned and shared with others, and establish the robustness and predictability of the technology. A number of unrelated but highly effective prototype interactive games were developed by the class. These examples provided evidence of the potential of the system and led to specific interactions used in "Journey ...." The results of the class at CMU led to the highly successful presentation incorporated with the Cell project.
Interactivity is a topic of great interest in education as well as entertainment, yet the challenges for educators are immense, and seldom is it examined with rigor. Knowledge of these issues has led to several new strategies in the Brain production, most notably, building a dedicated classroom for in-house research and formative critique and evaluation. In this way, we will place a much greater emphasis on evaluation methods in our pre-production and production work than in the Cell show, a tactic that we feel is essential to maximize the pedagogical impact of our production.
An important part of the next project about the Brain is the design of an interactive studio/laboratory in which development of this and subsequent similar projects can be carried out in an academic research environment without the constraints and limited access of the planetarium. This flexible lab will enable the project team to experiment with all aspects of the program, from audience size and segmentation to content presentation, length of program, navigational and narrational devices, scope of subject matter, visualization techniques and interactivity. In this setting can be tested audience receptivity as well as the technical coherence of the system, especially at points between linear and non-linear narrative. Perhaps most importantly, it will provide an opportunity to monitor and evaluate the audienceís receptivity to and understanding of the information presented; how well they are accomplishing the tasks assigned to them; how advanced is their progress. Evaluation experts working side by side with the showís creators will devise realtime in-line evaluation schemes
Conclusion
"Journey into the Living Cell" is regarded as a seminal project, representing a very high level of collaboration and invention in the emerging field of "edu-tainment". It broke new ground in the use of a planetarium for the presentation of material unrelated to space science, introduced to the general public an innovative audience interactive system, and presented to the public never before seen images from an esoteric but fundamental science. While it has been criticized for the overwhelming amount of information thrown at the audience, the show can be appreciated for its experiential beauty and lushness independent of its terminology and hard science. Immersion, sound, animation and entertaining interactive sequences transport the viewer into a new and different relationship with science than that ordinarily encountered in the classroom or even on Nova programs.
The task of creating this unprecedented production lay in the hands of many talented congenial artists and scientists. And it is a good thing that good humor prevailed since the task proved to be far more daunting than imagined. An artistic hand and philosophy guided the production, one that encouraged prototyping and experimentation, and maximized the application of the individual talents of the participants. After all, there were no models for us to base this experience upon and it was with great anticipation that we attended the first run through of the show to see what it was that we had created. Pressures of opening dates and last minute technical issues mandated our accepting what we saw and heard as "the show." But we also knew that since this was in part a digital medium, we could "tweak" the results and did so with some regularity in the months that followed.
There is little doubt that we will do the next production differently. The addition of an interactive studio/lab will mean that we can prototype until we are happy with the results and can test and evaluate with a real audience under realistic conditions on a continuing basis. This will mean that the show production will be much more akin to working in an artistís studio with the added benefit of being able to move at any time to the actual planetarium space a few miles away. Since the interactive medium is so young and undeveloped we expect that an extended period of experimentation will yield a much deeper understanding of its limits and potential and will stimulate unanticipated scenarios for its use. Developing the entire show as an interactive audio-video event will also dramatically affect the continuity and integrity of the production. "Journey.." was a hybrid between a traditional planetarium show structure and something as yet undefined, one of those strange mutations on the evolutionary chain. It worked but....
As the perspectives of the authors of this paper and of the show suggest, embarking on research in the area of interactivity is serious technical and conceptual business. Adding to the mix the desire to communicate scientific information and making the final production entertaining as well would seemingly impose so many restrictions and parameters that one would have to think twice about trying to create art out of this stew. In fact, as our clear enthusiasm and motivation indicates, all of us who worked on "Journey..." can hardly wait to begin the R & D for the Brain Project, so fertile does this groundbreaking work appear. For what we are doing is producing a new artistic medium, with full knowledge that we have stepped into virtually unexplored territory, and that we are defining a new state of art/science in the process.
Acknowledgments
This work was supported by National Science Foundation award #9355741 and by sponsors including the Buhl Foundation and Smithkline Beecham. The authors would also like to thank Carnegie Mellon University, in particular the STUDIO for Creative Inquiry, Center for Light Microscope Imaging and Biotechnology, and the Pittsburgh Supercomputing Center for their many contributions, and The Carnegie Science Center, home of the Buhl Planetarium, for its partnership in this endeavor. We thank the many individuals, too numerous to mention, at all of these institutions who gave us support, ideas, and encouragement. Animation software was developed by Karl Fischer and Brent Thomas. Andy Witkin offered guidance during early stages of development.
Begault, D. R. 1994. 3-D Sound for Virtual Reality and Multimedia. Cambridge, Massachusetts: AP Professional.
Bosi, M. 1990. "An Interactive Real-Time System for the Control of Sound Localization." In CMC Glasgow 1990 Proceedings. San Francisco: International Computer Music Association. pp. 112-114.
Chowning, J. M. 1971. "The Simulation of Moving Sound Sources." In Journal of the Audio Engineering Society, 19:2-6.
Dannenberg, R. B. 1993. "Software Support for Interactive Multimedia Performance." Interface Journal of New Music Research 22(3):213-228, August.
Dressler, R. 1966. "Dolby Pro Logic Surround Decoder Principles of Operation." Online at http://www.dolby.com/ht/ds&pl/whtppr.html.
Kendall, G. S. 1992. "Directional Sound Processing In Stereo Reproduction." In Proceedings of the International Computer Music Conference 1992. San Francisco: International Computer Music Association, pp. 261-264.
Perez, I., J. Arias, P. Fernandez, L. Perez. 1996. "Quad-Pan, spatialization system of music in live." In Proceedings of the 1996 International Computer Music Conference. San Francisco: International Computer Music Association, pp. 465-468.