\documentstyle[jair,twoside,11pt,theapa,misc]{article}
% They use
%\input epsf %% at top of file
% I use
\input psfig

\jairheading{?}{1994}{?-?}{\today}{?}
\ShortHeadings{NTD-Soar: An Integrated Model Agent}
{Nelson, John \& Lehman}
\firstpageno{1}

\begin{document}

\renewcommand{\grumble}[1]{\fixme {\tt (#1)}}
\newcommand{\mumble}[1]{{\tt (#1)}}

\title{NTD-Soar: An Integrated Model Agent}

\author{\name Greg Nelson \email ghn+@cs.cmu.edu \\
       \name Jill Fain Lehman \email jef+@cs.cmu.edu \\
       \addr Computer Science Department, \\
       Carnegie Mellon University,
       5000 Forbes Avenue,
       Pittsburgh, PA 15213-3891 USA
       \AND
       \name Bonnie E. John \email bej+@cs.cmu.edu \\
       \addr Human-Computer Interaction Institute,
             Computer Science Department, and
             Psychology Department, \\
       Carnegie Mellon University,
       5000 Forbes Avenue,
       Pittsburgh, PA 15213-3891 USA}

\maketitle

\begin{abstract}
An abstract should go here.
\end{abstract}

% Sections should be numbered and labeled.
\section{Introduction}
\label{Introduction}

For many years, work in psychology and artificial intelligence
has been evolving into a number of subspecialties that each focus on
particular abilities of human or computer intelligence, such as
vision, natural language, learning, and planning.  Each field has had
a different perspective and a different purpose, and significant
progress has been made in many of these areas.  Under the auspices of
cognitive science, the idea of integration has become widespread,
exhibited in cognitive models such as ACT-R\mumble{Anderson citation},
CAPS\mumble{Just \& Carpenter, 1987, or something else?}, the
Construction-Integration Model\mumble{Kintsch \& ???}, the Model Human
Processor \cite{CarMorNew83}, and Soar \cite{LaiNewRos87}.  

There have been different approaches to these integrative efforts.
Some systems, such as CAPS, have continued to focus on a particular
domain, and have achieved greater depth by integrating relevant models
and insights in that area.  Others, like ACT-R and Soar, have sought
broad coverage of human cognitive abilities, utilizing insights from
many specialized areas as guides in the construction of general
models.  However, a choice has often been made between depth and
breadth in these systems.  Recently, the desire to model real human
behavior in complex tasks has elicited a new sort of integrated model
-- a ``well rounded'' model that has the breadth needed for complex
human behavior and the depth required for believability.  

The NASA Test Director (NTD) performs a complex task of this sort,
coordinating the preparation of the NASA Space Shuttle before launch.
NTD-Soar takes the well-rounded approach in modeling the human NTD's
activities.  We have taken the approach of incorporating several
existing models which exhibit some depth in their particular areas
(language, attention, and problem solving) into a broad agent model.
Each of the independent capabilities has been validated using human
data, but none is sufficient to explain the range of behaviors
required by a human performing even a well constrained real-word task.
Combined, they create a model of a human agent, interacting with a
simulated physical world much like the real environment of the human
NTD, and exhibit behavior that a typical human might exhibit in the
same situation.  Our goal has been, in one sense, to develop a model
whose behavior is sufficiently plausible to enable it to pass an
NTD-specific Turing Test.  We can evaluate the plausibility of the
model by directly comparing its behavior to human performance data.

The NTD domain provides an excellent real-world arena for testing
integrated cognitive models.  The NTD performs a variety of activities
involving reasoning, verbal communication, and visual search.
Furthermore, the domain is laden with interruptions, task switching,
and apparently parallel activity, so any model must integrate the
capabilities in a way that allows for human-like facility with these
complex interactions.  Finally, a variety of data, some provided by
NASA and others collected independently, are available to evaluate how
plausibly the model behaves.

\section{The NTD Domain}
\label{The NTD Domain}

The NASA Test Director (NTD) domain requires an interesting variety of
cognitive capabilities.  Unlike a subject in a typical psychology
experiment, the activities of human NTD are widely varied and,
although a great deal of knowledge is required, much of what the NTD
does is very natural.  He reads, talks with people, makes decisions,
and is frequently interrupted by other people who need his attention.
In many of these ways, a model of the NTD domain is an interesting
model of human behavior in general.  However, it also has the
advantage of being fairly well constrained, so that the complexity of
what needs to be modelled is reduced.

The human NTD is responsible for coordinating many facets of the
testing and preparation of the Space Shuttle before it is launched.
He must complete a checklist of launch procedures that fills 3000
pages of looseleaf manuals (the Operations and Maintenance
Instructions, or OMI).  While launch preparations are underway, the
NTD talks extensively with other members of the launch team.  This
team of approximately 300\quest people communicates over a two-way
radio net called the Operational Intercommunications System (OIS).
The OIS has over 100 channels \grumble{number?}, many of which are
restricted to certain types of communications between certain {\it
stations} (individuals performing specific roles).  The NTD is
responsible for coordinating many of these individuals and for
tracking the progress of all major shuttle subsystems, through his
communications over the OIS.  This involves not only routine
management, but also the management of anomalies and the redirection
of problems to the individuals most capable of handling them.  The
domain is described in greater detail in \citeA{JohRemSte91}.

Much of the NTD's visual activity involves the OMI.  This checklist
consists of over ten thousand {\it steps}, each of which is assigned a
{\it sequence number}, in the {\tt SEQ} column.  Three of these steps,
with sequence numbers 16-0932 through 16-0934, are shown in
Figure~\ref{fig:omisamp}.  The majority of these steps are simply
communications about the progress of {\it procedures}, which are
substantially more complex and may involve numerous verbal steps as
well as actions taken by other members of the launch crew.  The steps
shown are an important part of a larger procedure that effects the
radio transmission (uplink) of the backup flight software (BFS) to one
of the shuttle computers.  Although only a portion of the steps are
used in any particular launch attempt, because many cover emergency
procedures and other ``anomalies,'' the NTD must still handle an
enormous volume of information.

\begin{figure}[htb]
\begin{center}
{\small
\begin{verbatim}
                        OPERATION INSTRUCTIONS

  SEQ       TIME    CMD  RESP            DESCRIPTION            
-------- ---------- ---- ---- ---------------------------------

16-0932             CVFS NTD  READY FOR BFS UPLINK
   
16-0933  -16M00S    NTD  *FLT PERFORM BFS PREFLIGHT UPLINK      *BCS
                              LOADING.

16-0934             *FLT NTD  BFS PREFLIGHT UPLINK LOAD
                         CVFS COMPLETE.
\end{verbatim}
}
\end{center}
\caption{A Segment of the OMI}
\label{fig:omisamp}
\end{figure}

As part of each step, there may be an associated {\tt TIME}, telling
when that step is expected to be performed.  The {\tt CMD} and {\tt
RESP} columns indicate the individuals involved with the step, the
former giving the person who is expected to initiate the conversation
and the latter the people who are supposed to be contacted.  The step
{\tt DESCRIPTION}, which is often read literally, contains the content
of that step.  A example of a typical visual search in the OMI
performed by the NTD is the {\tt FIND-NEXT-PARTICIPATION} ({\tt FNP})
task.  The NTD performs this task frequently (about 40\% of the tasks
during some parts of the launch are {\tt FNP} tasks).  Whenever this
task is selected, the NTD searches the ``CMD'' and ``RESP'' columns of
the OMI for his call letters (``NTD'') to locate the next step in
which he must participate.  At other times, the NTD is called upon by
other members of the launch team, and may need to locate a step on the
basis of its sequence number ({\tt SEQ}) or its description by
scanning through the book or possibly by remembering the location of
the step.

Communication about these steps takes place over the OIS.  Much of the
OIS communication that the NTD hears follows a stylized and restricted
sub-language, in other ways similar to English.  Some of the condensed
vocabulary has been popularized by citizens band radio: ``Roger,''
``Copy,'' etc.  The language is laden with acronyms, used by NASA to
refer to people, procedures, and equipment; these form another large
subset of the specialized vocabulary used in this domain.  Examples of
two dialogs involving the NTD, taken from audio tapes NASA recorded
during a normal launch attempt, are shown in Table~\ref{tbl:dialogs}.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{l l}
Speaker & Utterance\\
\hline
CVFS & NTD, CVFS. \\
NTD  & Go ahead, CVFS. \\
CVFS & Ready for BFS uplink. \\
NTD  & I copy. \\
\hline
NTD  & Houston Flight, NTD. \\
NTD  & Perform BFS preflight uplink loading. \\
*FLT & In work.
\end{tabular}
\end{center}
\caption{Sample NTD Dialogs}
\label{tbl:dialogs}
\end{table}

Each of these short dialogs corresponds to a single step in the
OMI manual (in this case, steps 16-0932 and 16-0933, shown in
Figure~\ref{fig:omisamp}).  In the first dialog, the CVFS
(the person at Cape Canaveral in charge of Vehicle Flight Software)
contacts the NTD, and indicates who is speaking.  This pattern, which we
call a {\it summons}, is expected in this domain, and
is necessary in part because many speakers may be sharing a single
radio channel, and no other identification (such as visual cues) is
available.  The NTD provides an {\it answer}, which, in accord with
the protocol, indicates who he believes is speaking.  Then the CVFS
provides the relevant content of his communication in an {\it
assertion}: he indicates that one of the shuttle computers is prepared
for the uplink.  The NTD indicates that he has received the
communication with the {\it registration} ``I copy.''  The next dialog
is a continuation of the same activity, in which the NTD directs
another member of the launch team (Houston Flight, or *FLT) to proceed
with the uplink.  At a later time (not shown), *FLT calls the NTD back
to confirm that this step is completed, so that the NTD has a record
that the entire procedure has been performed.  The NTD is responsible
for keeping track of launch progress by checking off steps in this
manual as they are completed.

NTD-Soar models the NTD's participation in these dialogs.  A variety
of components are needed for a system to model human performance in
this domain.  Much of the driving 
, looking in turn at the domain
model and the reasoning skills that are used in the domain, the Soar
architecture (which has a strong influence on other components), 
the natural language model, and the visual scanning model.  Because
two of the important components, language
and visual scanning, were created as parts of independent research
efforts, the discussions of these components will address the
background of these systems and the domains for which they were
originally built.

\section{Components}
\label{Components}

% Write about the components as they existed before integration.  Give
% lots of neato details about things like language data and attention
% experiments.  Then speculate on the extension needed when moving to
% the NTD domain, but don't go into detail.

The NTD-Soar model requires a broad set of human-like capabilities in
order to perform even the basic tasks of the NTD.  Rather than
developing a shallow model for each of these capabilities, which
include visual scanning, natural language comprehension and
generation, and learning, we have drawn upon extensive prior work in
each of these areas.  The visual processing capability was based upon
the work of NOVA \cite{Wie92} and expanded through
experimental work done as part of this project.  Our language model
derives from NL-Soar's systems for comprehension
\cite{Lew93b,LehLewNew91} and generation \cite{RubLeh94}.  
A learning mechanism called {\it chunking} is an integral part of the
Soar architecture \cite{LaiNewRos87}, and this provides a learning
capability for all of the other components.  Each of these earlier
models explores a particular area in depth, yet because each was
built within the common Soar framework, their combination into a broad
model was also possible.  Decision
making and problem solving knowledge in NTD-Soar came from an earlier
NTD agent that provided a functional account of behavior in the domain
\cite{JohRemSte91}.  This model was largely independent of the
architecture, and so will be discussed first.

% Subsections have this garbage around them, no label.
\vspace*{-0.05in}
\subsection{Task Knowledge}
\vspace*{-0.05in}

In their work, actual NTDs utilize a great deal of knowledge about the
shuttle, the support systems, and the launch team.  This resource is a
major component of the complete system, even though it barely touches
on the full range of knowledge a human NTD would have.  The knowledge
included in the current model covers basic communications,
page-scanning, and other routine knowledge, and certain specific
knowledge about particular steps and procedures.

The knowledge in the earlier NTD model was primarily of a general
nature, regardless of whether it concerned vision, language, or the
domain.  For example, the model was able to perform the {\tt
FIND-NEXT-PARTICIPATION} procedure.  This knowledge was represented in
the original model, but the detailed knowledge of OMI page layout and
visual search routines was not modeled.  Likewise, in the linguistic
domain, discourse conventions were followed even though no underlying
syntactic knowledge was used to support them.

Some domain knowledge characterized the relationships between tasks.
For example, when another launch-team member indicates to the NTD that
a step has been completed, an appropriate next action on the NTD's
part is to mark this step as complete in the OMI.  This level of
knowledge was specifically encoded in the original NTD model.  At
times, simply waiting (for information, or for an event) is the
appropriate action for the NTD, and other tasks must be postponed.

The process of integration is often (optimistically) viewed as a
simple wheeling up of existing components to create a combined model
with several capabilities.  In practice, even when a system has been
designed with divisions between the task and various components in
seemingly obvious places, the integration process may lead to
unexpected changes in this structure.  For capabilities involving
perception and action, the structure of the cognitive system and its
interface to this ``external'' world plays an important role in
defining the nature of the final model.
% An important observation is that often the
% seemingly obvious dividing lines between task and capabilities are
% changed in unexpected ways during the integration process.

\vspace*{-0.05in}
\subsection{The Soar Architecture}
\vspace*{-0.05in}

The Soar architecture provides a general structure for cognitive
processing shared by all the components of the NTD model.  This
framework includes a interface to perceptual and motor systems that
connect to a model of the external world; a common representation that
can be used for shared or private knowledge; principles for achieving
immediate, reactive behavior with expertise; and a learning mechanism
to compile knowledge into complex decisions and actions.

Cognitive models in Soar share a number of theoretically motivated
assumptions about cognition.  Knowledge in Soar is divided between
long term and short term memory.  The long term memories, or {\it
chunks}, encode problem solving experiences as well as facts.  The
short term memories, or working memory elements ({\it WMEs}), are
present for a limited duration; while they are present, they may
activate chunks to bring other knowledge out from long term memory
({\it LTM}) into working memory.  The contents of LTM may not be
deliberately inspected, while the contents of working memory are at
the very core of deliberation.

Many of the important facets of the Soar architecture are depicted
graphically in Figure~\ref{fig:ntdpsdgen}.  In a Soar model, different
bodies of long-term knowledge are organized into {\it problem spaces},
shown as triangles in the figure.  Each problem space contains a set
of {\it operators} (arrows between boxes) characterised by procedural
knowledge stored in LTM.  The operators modify, add, or remove WMEs,
typically WMEs that are part of {\it states} (boxes), representing the
current state of the problem solver.  The operators are the smallest
units of deliberate action, and must be performed sequentially.  At
any point, the knowledge required to unequivocally
select or perform an operator may be unavailable in the current
problem space; in this case, an {\it impasse} arises (thin dotted
arrow), and a subgoal is created to search for the knowledge in other
problem spaces.  When knowledge that is able to resolve the impasse
has been found, the subgoal is accomplished, and a new chunk is built.
Chunks represent generalizations of the problem solving in the
subgoal, and the process of learning new chunks is called {\it 
chunking} (wide dotted arrows).  Each chunk integrates knowledge
from one or more problem spaces and makes it directly available in the
problem space which gave rise to the subgoal; thus in similar future
situations, the system behaves {\it recognitionally}, that is, without
impasse.

The collection of active goals in cognition is ordered, from the
top-level goal down to the newest subgoal, into a {\it goal stack}.
As goals are resolved and new goals are created, this goal stack
shrinks and grows, as shown in Figure~\ref{fig:goalstackchanges}.
Every new goal is provided with a state, which may be a copy of the
state of the goal above (the {\it superstate}), or may be local to the
goal; this state will disappear again when the goal is resolved.
Operators typically work on this state, so that any changes
made to this state which are not propagated to the superstate will be
lost when the goal is resolved.

Soar makes a strong commitment to a serial decision-making bottleneck
in cognition.  Soar requires deliberate cognitive activities,
accomplished by operators, to occur serially.  Memory access, which is
not deliberate, can occur in parallel, and parallelism may appear in
perceptual and motor processing to the extent that the physical system
(such as the eye or the hand) allows parallel operations.
Furthermore, each cognitive operator has a fixed duration.  This
duration, estimated independently from the properties of the neural
system and from the general character of human behavior \cite{New90},
may fall within a range of about 30 to 300 milliseconds.  Although
this number is not fixed in the architecture, it has proven useful to
select a fixed size, 50 ms, to be shared by the Soar community.
Sharing this constraint with other researchers, including
\citeA{Joh88b}, \citeA{Wie92}, and \citeA{Lew93b}, has helped to
evaluate both the sequential operator model and the 50 ms duration.

%$$$$
\mumble{Kill this?}
One goal of this research has
been to show that a model exhibiting a serial cognitive bottleneck of
this sort is still capable of behaving in a realistic way.  Some
researchers
\cite<e.g.>{Fod83}\mumble{and Agre \& Chapman?} have suggested that a
model with  this structure is incapable of producing human-like
behavior in such domains as real-time speech comprehension and the
simultaneous processing of multiple senses.  By developing a model
that exhibits these sorts of behaviors in real time, we hope to dispel
the notion that serial symbolic models can only succeed in traditional
areas such as planning and problem solving.

\begin{figure}[htbp]
\centerline{\psfig{figure=ntdnl.gen.psd.ps,width=6.5in}}
\caption{A high level view of the NTD-Soar model, showing {\it problem
spaces} (triangles), {\it states} (boxes), {\it operators} (arrows
between boxes), {\it impasses} (thin dotted arrows), and {\it chunking}
(wide dotted arrows), and the connections between cognition,
the perceptual and motor systems, and the external world.}  
\label{fig:ntdpsdgen}
\end{figure}

\subsubsection{Modelling the external world}

As would be expected, the external world the Soar model perceives and
affects must lie
outside of the model of the agent.  Although it is possible to have
mental expectations about the behavior of the world, there must be a
truly external real world against which these expectations may be
tested.  Internally, Soar provides a few simple mechanisms for
interacting with this world.  The world may provide perceptual
stimuli, which are filtered through perceptual mechanisms and then
appear spontaneously in Soar's working memory as part of the {\it top
state} (see Figure~\ref{fig:ntdpsdgen}).  The perceptual filtering can
be an arbitrary transformation, but we have chosen a set of perceptual
mechanisms based on human perceptual parameters.  These mechanisms are
described later in the Integration section.  

Once in working memory, the perceptual elements can influence
behavior.  When cognitive processing wishes to initiate a motor action,
it again places information in specific parts of working memory,
connected to motor processors that translate the information into an
action in the external world, and invoke the external world simulation
to evaluate the results of the behavior.

An additional component provided by the architecture for use in the
world simulation is a simulated real-time clock.  Soar provides a
measurement of the time required for cognitive behaviors, but this
time may be shorter or longer than the time taken by the computer
hardware running the Soar model.  A plausible world simulation must
proceed at the pace of cognition, rather than the unrelated pace of
computer hardware time.  For this reason, the world simulation, as
well as the perceptual and motor systems, proceed at the pace set by
cognition. 


\subsubsection{Behavior in the top problem space}

Although cognition happens through the application of operators in
{\em all} problem spaces, 
the operators available in the top problem space are in some ways 
special.  Like all operators, they must have been learned at some
point, but once they are completely in the top space, there is little
additional learning to be done, and behavior will be as fast as it
ever can be.  These operators are characteristic of expertise in a
domain, whether it be a simple, knowledge-lean domain such as pressing
a button in a stimulus-response task, or a complex, knowledge-rich
task such as the NTD domain.
The {\it top state} associated with the top problem space is also
unique, because it provides the interface to perception and action.
Furthermore, this state is always present, and thus provides a means
for separate tasks to communicate.  Finally, specialized knowledge
used to mediate these activities and to guide task interleaving also
accumulates in the top problem space, either by design or through
learning.

The top problem space and top state are also special, because both are
always present, and because all 
interaction with the external world occurs through the top state.
Perception trickles in to this state from all of the various senses --
in the NTD model, only vision and audition are provided, but in principle
everything from smell to proprioception would arrive here.  Motor
actions are also initiated by placing certain motor commands on the
top state.  In both of these, a degree of parallelism arises,
since perception and action may proceed autonomously without cognitive
intervention.

\subsubsection{Learning and lower problem spaces}

Whenever knowledge in unavailable in a problem space, including the
top problem space, an {\it impasse} is created to search for the
necessary knowledge.  In response, a problem space is selected to
tackle the particular impasse or problem, and a 
Soar state is created within which to work on that problem.  
When the impasse is resolved, either because the knowledge
has been found or because another, more critical task is selected,
this newly created state is destroyed again, and any information from
that state that has not been preserved elsewhere (such as on a higher
state) is lost.  This lends extra importance to the top state, since
it is the only place to preserve information in working memory between
different subgoals.

The concept of problem spaces was originally motivated by the
observable difficulty people experience in moving from one
representation to another during problem solving \cite{NewSim72}.
\citeA{New90} hypothesized that the purpose of these different
representations was to limit the body of knowledge that had to be
considered at any particular point.  However, since no single
definitive model of the formation of problem spaces is available, most
problem-space based models leave the formulation of the initial spaces
in the hands of the model-builder.  Thus, the initial distinctions
between problem spaces within any particular model are often a
knowledge engineering issue, rather than a theoretical statement.  It
is necessary to provide Soar with a great deal of knowledge to solve a
particular problem, and the organization of this knowledge into
problem spaces makes the developer's task much easier.  As we will
show later, these initial distinctions are often blurred by learning
processes, so that eventually the organization of the knowledge in the
model is reshaped by the ways in which the knowledge is used.  This
reshaping can nevertheless create or leave barriers leading to the
observed difficulty of re-representation.


\vspace*{-0.05in}
\subsection{Natural Language}
\vspace*{-0.05in}

One of the NTD's most critical activities, and perhaps the one
involving the most interaction with the outside world, is verbal
communication with other members of the launch team.  Pattern-matching
approaches to this aspect of the model did not satisfy our desire for
cognitive plausibility.  Instead, we turned to a broad-based and
independently developed cognitive
model of natural language, NL-Soar.  The NL-Soar model was shown to
provide a new and highly successful account of both garden path
phenomena and unproblematic ambiguity resolution.  With the addition
of special vocabulary for the domain, essentially the same mechanism
was used as part of the NTD-Soar model.  

NL-Soar was designed to satisfy several psychologically motivated
constraints: immediacy of interpretation, interactive and modular
ambiguity resolution effects, garden path effects, unproblematic
ambiguities, parsing breakdown on difficult embeddings, and success on
acceptable embeddings \cite{Lew93a}.  Further development of
NL-Soar has continued in parallel with and has been informed by the
NTD project.  The current version of the model
has separate knowledge sources containing X-bar syntax, a model-based
semantics loosely tied to word senses taken from WordNet
\cite{MilBFGM90}, and skills for referent resolution.  As this
knowledge is applied, new combinations may arise.  Additional task
specific knowledge may be applied and incorporated as well.

Lexical access occurs spontaneously as words enter cognition.  Then,
one or more comprehension operators may apply to the word, placing it
into relation with earlier words and prior knowledge.  If no
comprehension operators are available, an impasse arises, and NL-Soar
is able to learn new recognitional comprehension operators.  Likewise,
if a choice of operators exists, an impasse arises again, and NL-Soar
learns which path to take in the future.  Because of the fixed size of
the operators (50 ms), and the known rates for human reading (200-300
ms/word), NL-Soar can only dwell on each word for a few operators on
average.  The more recognitional the knowledge becomes, the more
readily the model is able to meet this real-time constraint.
Typically, one operator is used to construct the syntactic structure,
as stored in the {\it utterance model}, and another to build the
semantic {\it situation model}.  Additional operators may be necessary
for reference resolution, repair after ambiguities, or shifts of
attention.

In the grey area between task and language lies the discourse model.
While moves in the discourse are accomplished through language, in
general it is the task the language is serving that governs the course
the discourse takes.  Thus, although a discourse model is sometimes
constructed by comprehension as part of the semantic interpretation,
it may also be augmented by the task.  In either case, it is utilized
by language generation to determine form and content of the next
utterance.

Generation also uses the situation and utterance models in its
processing, though of course information moves ``downstream'' from the
discourse model to the situation model to the utterance model.
Sometimes the discourse model may directly affect the utterance model,
when utterances convey a discourse move with negligible additional
semantic content.  (Many speakers, for example, say, ``How are you?''
as a greeting, without expecting an answer.)
From the utterance model, words go through lexical realization and are
moved to the motor system.  

In the process of generation, an important early step is the selection
of the next discourse move.  After each discourse move, one of a
number of options for the next discourse move are available (e.g
ignore, continue, close, accept, reject).  But while generation is
capable of implementing most of these in any particular circumstance,
it does not have the necessary knowledge to determine which of these
is appropriate.  This knowledge must in fact come from the task, which
determines the conditions under which each response is appropriate.
Because the impasse arises out of generation and the knowledge to
resolve it comes from the task, a chunk will be built which implements
generation in task-specific terms, leading to a {\it taskification} of
language.  Taskification occurs in other domains as well, as we will
see in the NTD visual scanning model.

In order to use NL-Soar as a component of the model NTD, a variety of
extensions were required.  Beyond the discourse knowledge just
described and the need for new lexical entries covering the NTD's
specialized sub-language, a perceptual model for audition was required.
This included, among other things, a phonological buffer which could
allow some delay before processing newly arriving words.  Furthermore,
the world model needed to be able to present the words of the other
speakers to the model at appropriate times.  On the generation side, a
motor system had to be developed that was capable of cuing a sequence
of words in an articulatory buffer.

\vspace*{-0.05in}
\subsection{Visual Scanning}
\vspace*{-0.05in}

Another important component of the NTD's work is the use of manuals
and charts providing information about the sequence of the launch. 
Rather than making this high-level conceptual structure available
directly, we chose to draw on well developed mechanisms of visual
attention and to build up the necessary conceptual structures in
cognition as part of the NTD model.  To
provide this component of the model, we turned to NOVA, a model of
covert visual attention developed by Mark Wiesmeyer \citeyear{Wie92}.
{\it Covert} visual attention, as he describes it, is the ``variety of
visual attention that does not involve overt movements of any sort,''
such as head or eye movements.  NOVA was designed as an extension of
the Model Human Processor, and was implemented in Soar5.  It provides
accounts, including fits to human timing data, for a number of
different types of attentional phenomena, such as pre-cuing
\cite{ColHofEri73}, crowding \cite{LaBBro89}, decay \cite{Spe60},
illusory conjunctions \cite{TreSch82}, search \cite{TreGel80},
counting \cite{ChiKla75}, and detection \cite{HugZim85}.

NOVA presumes that visual attention must precede further cognitive
processing of the visual stimuli, and places visual attention under
deliberate cognitive control, mediated by operators.  An important
component of the model describes how the attentional spotlight or
``zoom-lens'' determines the features available to cognition.  The
model further enumerates the set of cognitive operators for adjusting
and moving this attentional spotlight, and for processing the visual
stimuli once they are present in cognition.  These operators are
divided into visual operators and semantic operators.

The visual operators respond to items in the ``visual image store.''  
{\tt ATTEND} operators are used to shift visual attention.  These
operators may be created in parallel for all unattended features
present in the visual store.  They may be ignored, or one of them may
be selected to shift attention to a particular feature.  The {\tt
RECOGNIZE} operator is used in identification.  When features arrive,
color, shape, and other components are separate.  {\tt RECOGNIZE}
joins these features together for a particular object, so that
conjunctive tests (e.g. ``red and Q-shaped'') can be performed on the
item.  The semantic operators {\tt VERIFY} and {\tt RESPOND} are then
used to achieve the demands of the task.

To use NOVA as part of the NTD model, several components needed to be
fleshed out for the particular task.  The OMI pages needed to be made
available as raw perceptual stimuli, and the visual operators relevant
to the NTD's task needed to be selected at the appropriate points.
Furthermore, because the OMI extends substantially outside the human
NTD's fovea, the model needed to be extended to include overt
attention, including both eye and head movements with corresponding
timing models.  The approach taken for all of these extensions, to the
task and language as well as vision, is described next.

\section{Integration}
\label{Integration}

Bringing together this collection of components only satisfies some of
the requirements for the integrated system.  There are a number of
outstanding problems which must be solved by adding knowledge to the
existing components, such as task-specific knowledge for language.
Other problems must be addressed by adding knowledge between the
components, to specify, for example, how the components may interact.
Finally, some problems may only be resoved by constructing new
mechanisms that extend the architecture, including perceptual and
motor mechanisms that are not yet an integral part of Soar.  Before
looking at how these problems are resolved, let us turn again to the
goals of the integrated system to understand which ones have already
been addressed by the collection of components, and which ones still
remain to be solved by changes and additions to the components.

\mumble{This list probably needs to be expanded, and needs to refer
back to previously described goals, which haven't been laid out in the
earlier sections as they should have been.}

The model derives a collection of domain expertise from the earlier
NTD domain model.  This includes knowledge about different tasks, such
as visual searches and verbal communication, and determines which
tasks are appropriate at a particular time.  However, this model does
not provide any of the details of page layouts, or the linguistic
knowledge necessary to comprehend and generate the unusual utterances
of the domain.  It also provides only sparse information about how the
different components should relate, so additional detailed knowledge
about component interaction and interleaving is required.

The NL-Soar language model provides a good general model of language,
but must be extended to cover the vocabulary of the domain.
Furthermore, the existing attentional model in NL-Soar is adequate for
reading, but insufficient to handle spoken language, for which the
listener has little control over timing and speech rates.  In
consequence, the model must be extended to include buffering (for
cases when speech arrives before it is expected) and interruption
(when speech arrives faster than language processing can occur.)

Covert visual attention, as modelled by NOVA, is only applicable when
the next item to receive attention falls within the fovea.  This model
must be extended to provide ``overt'' visual attention, since many of
the attention shifts the NTD makes in scanning a page involve eye
movements.  Also, as just mentioned, an attentional model is needed
for audition.  Although NOVA was designed for a different modality,
extending it in this direction seemed the most parsimonious way of
filling this requirement.

With these additions, we complete the set of requirements we set forth
for a model of the NTD.  Next we discuss in detail each of the
extensions and modifications to the components and how they achieve
the requirements.  Once the components have been fleshed out in the
required ways, we will turn to the more global issues of component
interaction.

\vspace*{-0.05in}
\subsection{Domain-Specific Component Knowledge}
\vspace*{-0.05in}

NL-Soar's default linguistic knowledge is primarily about such
entities as dogs, horses, barns, and blocks, and actions such as
racing, falling, moving, etc.  This sort of knowledge was sufficient
to demonstrate syntactic diversity, but must be extended when any more
specialized domain is required.  The new knowledge about the domain
includes lexical entries for a number of NASA specific words, such
as ``NTD,'' ``uplink,'' and ``copy.''  These entries provide some of
the knowledge about the constructs in which the words may sensibly
appear, and in some cases knowledge about the meaning of the word.
(Some of this knowledge is used, for example, to allow the model NTD
to recognize that the call letters ``NTD'' refer to itself.)
This sort of knowledge also tells how to say people's call letters:
``*FLT'' on the page is translated to ``Houston Flight'' in
conversation. 

The knowledge that communication with someone is the way to perform an
OMI step is also necessary.  An important piece of knowledge along
these lines, alluded to earlier, is the
domain-specific mechanism for selecting discourse moves.  In the most
typical situations, the model is always required to accept a new
conversation (this is what the NTD's training expects, and fits the
behavior we have studied) and continue the current conversation rather
than starting a new one.  Furthermore, to perform a summons, an
answer, or a registration in the NTD domain, which are specific
instances of general discourse moves, specific linguistic forms
peculiar to this domain must be selected.  Answers, for example, can
have any of the forms listed in Table~\ref{tbl:ansforms} (where ICP is
the {\it initiating conversational participant} and OCP is the {\it
other conversational participant}).

\begin{table}[htbp]
\begin{center}
\begin{tabular}{l}
Go. \\
Go ahead. \\
Go ICP. \\
Go ahead, ICP. \\
OCP, ICP.  Go ahead. \\
OCP, go.
\end{tabular}
\end{center}
\caption{Some forms for an ``answer.''  The ICP initiated the dialog
with a summons; the OCP is the speaker for the answer.}
\label{tbl:ansforms}
\end{table}

Although NOVA doesn't need to know much about the organization of the
page, efficient task performance using page scanning abilities depends
on this sort of knowledge of low level details.  Information such as
where the head needs to turn to find the OMI or countdown clock (CDC),
the location of rows, columns, and page numbers in the OMI, and how to
read the CDC time is required.  Within a step, knowledge about which
elements are necessary for which procedures tells the model such
things as where to look to find out if a step is complete (SEQ column)
and how to evaluate whether to perform a step now or later (checking
the TIME column, and looking back to previous steps if no time is
listed).

Finally, a limited amount of domain knowledge was added to the system
to allow it to cope with certain specific situations. 
When searching for a specific step,
knowledge about where it occurs in the launch may help guide whether
OMI search proceeds forward or backward from the current point.
For reasons described in the discussion of
performance below, there is also some very advanced expertise that
allows the NTD model to initiate the summons to Houston Flight shown
in Table~\ref{tbl:dialogs} {\em without} looking at the OMI for this
step.

\vspace*{-0.05in}
\subsection{A Detailed Model of the Environment}
\vspace*{-0.05in}

In order that the agent be as plausible, and the simulation as
realistic as possible, we developed a detailed simulation of the NTD's
work environment, including a simulated OMI, simulated OIS
communications, and a countdown clock on the wall.  Many other
elements of the environment, including the people located in close
physical proximity, were ignored.

The pages of the OMI, though encoded electronically, were presented as
arrays of text characters with no further structural markings; it was
the role of the modeled perceptual system to capture the structure of
words, rows, columns, and pages.  We developed a
perceptual mechanism to group sets of characters bounded by blank space
into basic perceptual units.  These units generally correspond to
words, but are not conceptualized or decomposed at this point.
Each unit is composed of a shape feature and a color
feature; most of the pages of the OMI are black text on a white
background, but the presence of the color feature makes it possible to
study some of the same experimental conditions tested with NOVA
without modifying the perceptual mechanism.  One may shift attention
to any of these units, but unless they fall within the
fovea,\footnote{Our fovea is defined generously as including the 
entire region that appears to affect reading rate, from four
characters left of fixation to fourteen characters right of fixation
\cite{RayWelPol80}.} its shape cannot be resolved adequately, so an
eye movement is required if this word is to be read.

Likewise, the OIS communications are pre-encoded into words; no speech
recognition is performed, although some consideration has been given
to this issue.  As a simplification, the words presented on the top
state auditory input are lexically encoded and are separated even if
they are coarticulated.  Speaker information is available as well, as
a separate feature -- in human perception, this information might come
from an analysis of particular acoustic characteristics of the
individual.  The time course of the words is matched to that of the
original tape recordings provided by NASA.  For the purposes of
simulated input to the NTD, only the items spoken by others are
presented, and their timings are specified on a word by word basis, as
in Table~\ref{tbl:dialogdetail}.  Because of the selected level of
representation, comprehension is able to proceed in the same manner as
it would were the words printed instead of spoken.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{r r r l l}
Start & End & Speaker & Word	\\
\hline
148357 & 148829 & CVFS & NTD	\\
148829 & 149502 & CVFS & CVFS	\\
150814 & 151044 & CVFS & Ready	\\
151044 & 151211 & CVFS & for	\\
151211 & 151566 & CVFS & BFS	\\
151567 & 151895 & CVFS & uplink
\end{tabular}
\end{center}
\caption{Detail of NTD Dialog}
\label{tbl:dialogdetail}
\end{table}

The motor system is relatively simple, allowing eye movements within
the OMI pages, head movements between pages and between the OMI and
the CDC, hand movements to turn the page, and speech.  Each of these is
initiated by an {\tt INTEND} operator which places a motor command on
the top state.  The duration for each action is calculated from human
performance measurements or estimates.  The eye movement, for example,
has a duration that is a function of the distance as estimated from the
results of \citeA{Wes54} and \citeA{Fuc76}.

One important characteristic of the motor system is that once an
action is initiated, it proceeds automatically without conscious
control, although it may be possible to interrupt the action with
another deliberate act.  In speech, once the word has been released
from cognition, control is lost and the word will be articulated in
the order in which it was released.  Although articulation may in
principle be halted, the words already released may not be modified
after they have left cognition.  Articulation errors (and similar
errors in other modalities) may still occur after this point as well.

\vspace*{-0.05in}
\subsection{Component Interactions}
\vspace*{-0.05in}

% $$$$
Because of the similarities between immediate behavior and domain
expertise, we have found it useful to categorize the
top-space NTD-Soar operators in much the same way Newell did in his discussion
of immediate behavior in Soar \citeyear{New90}.  Although the NTD's
activities are not strictly ``immediate behavior,'' these categories
still characterize well several important types of cognitive
activities the NTD performs:

\begin{description}
  \item[Attention] operators direct the perceptual mechanism to
adjust the type and the volume of information provided about
particular elements in the sensory input.
  \item[Comprehension] operators interpret and elaborate the
perceptual inputs in conceptual terms, or perform further inferences
on the conceptual representation.
  \item[Task] operators select the next task to be performed, changing
the state to control the selection of other operators.
  \item[Intend] operators initiate physical responses from the system,
such as simulated speech or hand movements. 
\end{description}

All tasks are performed by collections of these types of operators,
plus the (non-cognitive) perceptual and motor operators that interact
with the physical world.  Naturally, language comprehension is
performed by perceptual and comprehension operators, and language
generation leads to intention and motor operators.  Processing of the
visual page is also performed through similar operators -- intention
operators produce eye movements, and perceptual, attention, and
comprehension operators search for information and perform the visual
pattern matching that allows NTD-Soar to determine whether it has
found the information it is seeking.  These categories lead to a
natural organization of knowledge into problem spaces: problem spaces
for suggesting new tasks, for selecting new operators for particular
tasks, and for mediating between operators within and across tasks.
% $$$$

Using PEACTIDM as a constraint on the integration has a number of
obvious ramifications for the control structure.  Each capability
will, after learning, lead to a collection of top-space operators.  It
must remain possible to compose these operators into intricate,
knowledge-rich behaviors, so the knowledge required to mediate among
them must also be available.  Because only one operator may be
selected at any time, we must be able to fluidly switch between the
capabilities, interrupting and resuming if necessary, in order to use
them in concert.  There are also some deep connections between the
choice of PEACTIDM and the style of communication between the
previously independent capabilities.

\subsubsection{Control and Communication}

In our view of PEACTIDM models, the Tasking operation is the primary
means for controlling behavior.  A Tasking operator selects a
particular task to be accomplished by the operators that follow.  This
task might be very simple (as in the case of immediate behavior) or
highly complex (when a difficult problem is set up to be solved
mentally).  However, in either case, the operation remains the same --
to place the necessary information about the current task into Soar's
working memory, whence it can guide problem solving by controlling the
selection of operators.  In particular, we decided to make this
information available on the top state.  There, along with perceptual
information, it has dominion over the remainder of the system.

This choice has several consequences.  Because operators place
information on the state that can only be removed by another operator,
a task remains selected until a deliberate choice is made to remove
it, because of success or failure, or to replace it because another
task becomes more important or more interesting.  From this point,
however, the task selection can guide the operators selected in the
top space as well as in lower spaces, which is important for achieving
PEACTIDM-style immediate behavior.  In other words, in order for
a Tasking operator to control immediate behavior, it must control the
operators which appear in the top space.  But since the Tasking
operator cannot remain in place while other operators are selected
(because of the serial bottleneck), it must record the selected
task in a way that can effect later operators in the top space.  The
only place such information can persist for an arbitrary period of
time is on the top state.

Most tasks, including visual scanning tasks such as {\tt FNP} and
language generation tasks, and composite tasks such as {\tt
PERFORM-STEP} (which involves reading the step, summoning someone,
etc.), follow a Tasking-operator based task-selection model. 
Typically only one of these tasks is selected at a particular point,
because the NTD's duties specify which should have priority.  Tasks
are selected based on the NTD model's knowledge of the state of the
launch, and the perceptual inputs the model is receiving.  When a task
is completed, or when information is perceived which gives a new task
higher priority, another Tasking operator selects a new task.
Finally, when multiple tasks are relevant simultaneously, it is
necessary to negotiate among them, providing further knowledge that
prioritizes them or interleaves them.  This sort of knowledge can be
very simple, and is used extensively when visual tasks and language
comprehension coincide.  More extensive research on this has been done
\grumble{anything published} by Gary\quest and Phoebe\quest.

The ``task'' of comprehension has a special status in the model,
because Comprehension is itself a basic PEACTIDM operation.  While a
task such
as comprehending spoken words can be selected and performed in the
same manner, often it is necessary to perform some comprehension
simply to decide whether a conversation requires further attention.
For this reason, the NTD model makes it is possible to perform some
amount of comprehension as part of almost any selected task.
This model for comprehension is in fact just a special case of
parallel or contemporaneous tasks.  Note that Soar makes no commitment
about whether two or more {\em tasks} may be selected in parallel,
insisting only that the cognitive operators implementing these tasks
occur serially.  However, in any case that tasks might co-occur, some
form of knowledge, either specific or default, must be present to
determine the selection of a unique operator at any given instant.
Specific knowledge is used to give comprehension high priority over
other tasks.  In practice, because there are often pauses in
conversation, and because routine comprehension is relatively fast
compared to articulation, there typically little contention between
task and comprehension.  When no words are available, the task simply
proceeds unimpeded; when words are present, language takes over
temporarily.  Further knowledge expresses the relative priority of
comprehension and generation (comprehension is more important unless
the NTD has been waiting too long without hearing something, in which
case he gets ``impatient,'' and proceeds with generation).

Of course, the integration would hardly be useful if the different
capabilities were unable to communicate effectively.  Because the
different capabilities typically are not active at the same time,
communication is made more complex, involving the passing of
information over time as well as across components.  The organization
of the system around a central top space that mediates between other
tasks lends itself to a shared representation.  The top state holds a
global ``situation model'' (cast in the same form as that of language,
because this representation was well developed) that represents visual
objects, task knowledge, and the discourse state.  Each capability
that needs to extract information to guide its behavior may find that
information in this model; each capability that wishes to share
information adds it to this model as well.

Having discussed the composition and static character of the model, we
move on in the next section to its dynamic character.  Specifically,
the process of interleaving different capabilities makes it necessary
at times to delay, interrupt, and resume behaviors, introducing a
number of novel issues into the model.

\subsubsection{Delay}

A need to delay processing for a task occurs when the task in question
has a
lower priority than the task currently selected.  In some cases, the
delay may be as short as the completion of the current operator, but
in others, a larger amount of work might be needed before the selected
task can relinquish control.  In these cases, it is important that the
task that has been delayed can still be performed, if necessary, when
the delay is over.  An important example of this is the processing of
spoken language.

In the absence any sort of buffering, an agent involved in any other
action when a word was first heard would never be able to process that
word.  This is true even if the agent's prior activity {\em was}
language processing -- the next word might be missed while processing
the previous one.  To prevent this sort of failure,
the words filter into a phonological buffer with a limited capacity
of 2 seconds and a short decay time, also 2 seconds. (Baddeley,
\grumble{Need reference to Rick's book, I think.})
As in vision, the words only appear in working memory if attention has
been paid to the auditory channel; otherwise, the presence of speech
is noted, but the words are not available for lexical access.  (This
is a simple extension of the NOVA model to the auditory domain.)
Once attention has been focused on a particular word, subsequent words
are automatically attended for some period of time.  When the word has
been attended lexical access proceeds automatically.

Because of the limitations of the phonological buffer, comprehension
must act quickly on spoken material, before the speech decays from
memory.  Nonetheless, it is possible to postpone comprehension, or even
postpone attending to the speech, because of this buffer.  Although a
temporal buffer for visual perception is possible as well, its
buffering effect is not critical as long as the OMI page does not
vanish before the NTD can read it.

\subsubsection{Resumption}

When one task is interrupted by another of higher priority, it is
often necessary to return to the first task.  Although it is sometimes
possible to simple start the first task from scratch, the possibility
of frequent interruptions makes this undesirable.  It is better for a
task to be able to make some amount of progress before interruption,
and then {\it resume} its work where it left off.  This sort of
resumption after an interruption is, in some ways, a special case of
communication between tasks -- in this case, a message is being passed
from the older instantiation of a task to some future instantiation of
itself.

When a task is selected that requires deeper problem solving,
involving impasses and subgoals, new states are created for the new
subgoals.  These states may be simply ``blank slates,'' or may contain
extensive information copied from the state above (super-state) or
from other sources (higher levels, perception, operators, etc.)  These
states serve as working memory for the current problem-solving effort,
and disappear when the impasse is resolved.  If, however, a solution
to the problem or other useful information is discovered, this
information can be passed back to the super-state or to the
super-operator, which results in the formation of a chunk.  Because of
this mechanism, it is easy to learn operators which add information to
or change the shared situation model.  Any time a problem space has
information it wishes to share, it may add that information to the
situation model; if the problem it is addressing is part of a
higher-level operator, this operator will then be able to make the
same additions without the impasse. 

The {\it grain-size} selected for the task, the quantity of work that
it performs before it returns a result of this nature, is a critical
determinant of the ability for resumption.  If a large amount of
problem solving goes on in the subgoal before a result is returned,
there is more time, and therefore a greater chance, for an
interruption before any result is available.  Without this
intermediate result, the problem-solving must start from scratch the
next time.  In contrast, the sooner the result is returned, the
further the problem-solving will be able to proceed the next time.
And, as an added benefit, whenever a result is returned, a chunk is
built that moves this task further toward expert performance in the
future.

\section{Performance}
\label{Performance}

In order to establish that our model really performs like a human NTD
on the tasks we consider, we must compare it with detailed human data.
First, we look at the language performance, in comparison to recorded
dialogs of the human NTD.  The speech data available from the NASA
tapes was one of the major sources of information about the NTD's
behavior, and we look in some detail at the model's performance in
mimicking certain dialogs.  Then we address the visual search behavior
at a high level, comparing the human and model response times after a
lengthy search task.  We look at a particular search through the OMI
that seems necessary to explain the NTD's communications.  Then, we
look at preliminary comparisons between the model's predicted eye
movements and human eye movements while scanning the OMI.  Finally, we
show briefly that the capabilities of language comprehension and
visual attention brought to the integration by the NL-Soar and NOVA
models can still be used by the model NTD despite his domain
expertise.

\vspace*{-0.05in}
\subsection{Dialogs}
\vspace*{-0.05in}

Given dialogs like the ones shown in Table~\ref{tbl:dialogs}, an
obvious goal was to reproduce the same utterances as the NTD produces,
given the same context and the same perceived speech from other
members of the launch team.  Rather than stopping here, however, we
sought to reproduce not only the form, but the {\em time course} of
the dialogs.  Although we did not attempt to fit the characteristics
of the particular NTD in the recording, we wanted the timing of the
responses to be plausible, so that another person interacting with our
model NTD might believe the model was human.  Again, the seriality and
known duration of operators (in NTD-Soar, 50 ms each) are particularly
important assumptions for modeling the time course of behavior,
because they make it possible to evaluate the duration of cognitive
activities. 

\begin{figure*}
\centerline{\psfig{figure=tl-uplink-dialogs.ps,height=9in}}
\refstepcounter{figure}
\label{fig:tl-uplink-dialogs}
\end{figure*}

A presentation of the model's performance, for the same ``BFS uplink''
dialogs, is shown in Figure~\ref{fig:tl-uplink-dialogs}.  This figure
shows a segment of the full time-line that would begin with the
beginning of the transcribed audio tape.  Several different categories
of events are plotted in rows, all relative to this time axis.  First,
the transcribed words of both the NTD and other speakers, like those
shown in Table~\ref{tbl:dialogdetail}, are presented as a basis for
comparison in the dotted boxes of the ``observed speech'' category.
Below this, the model's behavior is shown in some detail.  The black
boxes mark the times at which the model's perceptual mechanisms are
processing the words of the other speakers.  Each of the tiny grey
boxes represents one 50 ms cognitive operator, grouped in this figure
into behavioral categories of language comprehension and generation
and page scanning.  Finally, the white motor operator boxes show both
the timing of the model's generated responses and the eye movements
occurring during page scanning.

The fit between the human data and the model performance is reflected
by the dashed lines connecting the two.  By design, the perception of
spoken words coincides with the time they are uttered.  In this way, a
comparison between the human and model responses is a comparison
between the human cognitive processing time and that of the model.
For immediate responses to the CVFS utterances, the model responds
slightly later than the human, but well within the range of individual
variation.  Some of the differences are probably a result of delaying
cognition until the articulation of each word is complete -- when
hearing ``CVFS'' it is unlikely that an NTD with extensive
experience would not already have a good idea of the speaker earlier
in the articulation.  In the same vein, an experienced NTD would
recognize ``NTD, CVFS'' as a summons long before the second set of
call letters was completed, and this is all the information necessary
to make the subsequent ``answer'' discourse move.  Despite the model's
conservative assumptions about these factors, the response comes in
about 350 to 400 ms, which compares favorably to Norwine and Murphy's
\citeyear{NorMur38} mean of 410 ms for responses in telephone
conversations.

% Calculated from trace: st(Go) - et(CVFS)  = 149850 - 149502 = 348
%                        st(I) - et(uplink) = 152300 - 151895 = 405

After summoning Houston Flight, the NTD pauses (presumably waiting for
an answer) before continuing and generating the next utterance (a {\it
direction}) himself.  The length of this pause is modeled as the
expected time to hear a response from the other speaker (again, 410
ms) plus an additional operator to initiate the next utterance if it
is not heard.  This seems to fit the observed human performance, a
delay of 480 ms, fairly well.

% Calculated from trace: st(Perform) - et(NTD)  = 153969 - 153489 = 480

Finally, despite the procedures that the NTDs are trained to follow,
we believe that the human NTD must be initiating the second step
(summoning Houston Flight) {\em without} checking off the prior step
or consulting the OMI for information on the new one.  The reason for
this that there is literally {\em no} time between the previous
utterance, ``I copy,'' and the new one, ``Houston Flight,
NTD.''\footnote{In fact, the words ``copy'' and ``Houston'' are
actually co-articulated, sounding like ``copyouston.''}  This leaves
the NTD no time for writing or for turning pages to find the right
spot.  The human NTD may be performing this behavior in either of
two ways.  It is possible that he has already found the location and
read the steps, and so has them in memory.  Another possibility is
that he is simply performing the second step from memory.  These
alternatives are impossible to discriminate given the data available,
since either might explain the extremely rapid turnaround observed.
We have chosen to model the second, since we believe that the NTD's
expertise in this critical launch step is probably sufficient, given
enough context, to recall who is responsible for the next step without
consulting the manual.  \grumble{It would be nice to be able to say
that we had actually done the other as well and the results were
similar.  I might be able to do this, if we think it's useful.}

The same kind of gross timing comparisons can be applied to the visual
search model, despite the fact that it makes much finer predictions at
the level of individual eye movements.  The timing comparisons
explored in the next section rely on the same audio data for the human
performance measurements.  After this, a set of exploratory data on
human eye movements are used to evaluate the finer grained model.

\vspace*{-0.05in}
\subsection{Visual Search}
\vspace*{-0.05in}

Another important evaluation is that of the page scanning behaviors
that the NTD uses to find information in the OMI.  Initially, we
compared these behaviors only to the same audio transcript.  In one
particular set of dialogs, shown in Table~\ref{tbl:fnpdialogs} the NTD
completes a previous step, pauses for an extended period, and then
initiates a new step.  The new step is one which lists the NTD call
letters in the RESP column, so it is reasonable to believe that the
NTD finds it in by performing a {\tt FNP} search,
and this is what our model does.

\begin{table}[htbp]
\begin{center}
\begin{tabular}{r l l}
Time (ms) & Speaker & Utterance\\
\hline
11902 & NTD & Step 907 verified complete. \\
13357 &     & {\it (7236 ms pause)} \\
20593 & NTD & Houston Flight NTD \\
\end{tabular}
\end{center}
\caption{Sample NTD Dialogs}
\label{tbl:fnpdialogs}
\end{table}

\begin{figure*}
\centerline{\psfig{figure=tl-fnp789.ps,height=9in}}
\refstepcounter{figure}
\label{fig:tl-fnp789}
\end{figure*}

Another time-line comparison of the human and model behavior is shown
in Figure~\ref{fig:tl-fnp789}.  Here no auditory perception or
comprehension is involved.  Comprehension of the page must occur,
however, at a deeper level that the superficial pattern matching that
was going on in simply finding the letters ``NTD.''  Also, because
this particular search requires a page turn, we see a hand movement.
During this time, cognition is idle for an extended period because the
visual information from the page is unavailable.  If another task were
pending, this time could be used for interleaving instead, but during
this step there was no external indication of a second task.  Once the
step is found,
the NTD model notes various information, checks the CDC time (looking
up from the manual and back again), and determines that the step is
ready to be performed.  This leads the model to generate a summons for
the other party involved in the step.

In this case, the model's prediction falls dramatically short of the
time the human NTD took.  Of course, we can only guess at what, if
any, other activities the human NTD might have been performing during
this 
time.  However the non-NTD subjects in the eye-tracking experiments
discussed next, who had no other distractions, took an average of over
9800 ms for the same search, slightly longer than the expert NTD.  The
important thing to observe is that, in stark contrast to certain
claims about the limitations of serial cognitive models, we have {\em
plenty} of time to perform the necessary activities, even though very
little cognition goes on during the lengthy pauses for page turns and
head turns.

Even though the NTD model is faster than the human, it is important to
evaluate these differences in other ways because of their magnitude.
Evaluating the predictions of individual eye movements seemed a good way
to look for clues to where the model was going astray.  In fact, as we
shall see in the next section, the human data we collected suggest
that the model may be on the right track with its visual scanning
model after all.

\vspace*{-0.05in}
\subsection{Eye Movements}
\vspace*{-0.05in}

During the development of the model, we were curious about the
accuracy of our eye-movement predictions.  Unfortunately, NASA felt
that video cameras pointed at the NTD were too intrusive for the
section of the launch we were studying, and traditional eye-tracking
equipment is far more intrusive than a camera.  After we had completed
the model, a new, non-intrusive, neural net based gaze tracker
\cite{BalPom94} became available at CMU, and so we ran a small
eye-movement study to evaluate the model.  Although we did not have
the resources to run a full-fledged study, we were able to draw some
valuable comparisons, looking at both the time course and the
locations of predicted and actual eye movements.

Four subjects participated in the study.  Two were faculty in the
Computer Science Department at Carnegie Mellon University (CMU), while
one was a CMU graduate student and the other a University of
Pittsburgh professional student.  While the actual NTDs were not
available to participate in such experiments, all of the subjects
except the last have been involved with the NTD project for at least
two years.  The last subject had some knowledge of the project, and a
small amount of tutorial instruction about the OMI document.  The data
were recorded using a piece of software written for this experiment,
which invoked the gaze tracking software.  The hardware used was a Sun
SPARC 10 with special purpose video input hardware.

Samples of the eye position were taken at a roughly fixed interval.
Each eye location, key press, and instruction presentation was marked
with a time-stamp (in milliseconds) and recorded in a file for later
analysis.\footnote{The actual sampling rate for eye position
measurements was variable due to slight changes in system load, but
stayed fairly close to 50Hz.  The eye-tracker is known to operate more
slowly than this, at about 15Hz.} The text of the OMI was manually
transcribed and presented as black characters on a white background,
in a font size very closely approximating the font used in the actual
document.  Subjects were seated 50 to 70 cm from this display.
Because the OMI was electronic rather than paper, subject key presses
were used to turn pages, and only one page (rather than a pair of
facing pages) was presented at a time.  To compensate for this, a
brief pause was inserted between the key press and the display of
the next page, to correspond to the delay required for the physical
movement of the page.

\begin{figure*}
\centerline{\psfig{figure=tl-fnp792+data.ps,height=9in}}
\refstepcounter{figure}
\label{fig:tl-fnp792+data}
\end{figure*}

In evaluating the match between the time course of the model and human
behavior, we selected a reasonably long task including different sorts
of behavior.  In particular, another {\tt FNP} task, this one
searching for the same step introduced by the familiar dialog of
Table~\ref{tbl:dialogs}, provides a series of model eye movements and
a page turn that can be compared with those of several subjects.
Another time-line comparison is shown in
Figure~\ref{fig:tl-fnp792+data} showing two datasets from one subject
and one each from two others.  The time-lines are synchronized at the
start, and the pauses for page turns are adjusted to be equal in
duration to those used by the model.  In this diagram, the only
comparisons drawn are at these page turn points, but even from this it
is clear that the model's behavior falls within same range as the
performance of the subjects.

This suggests that the differences seen in the previous section,
in which the model dramatically out-paced the human NTD, might in fact
not be due to gross errors in the visual scanning model.  Instead, it
seems more plausible that the extra time taken by the human NTD
results from other tasks that he may be performing simultaneously,
such as crossing out completed pages.

The model also predicts individual fixations which appear rather
similar to those of subjects, at least at a gross level.  Because the
eye tracker training and configuration was not optimal, we were unable
to achieve the full resolution the eye tracker appears to be capable
of.  However, the roughly 4$^{\circ}$ accuracy we did achieve made it
clear that the subjects were focusing their searches in particular
columns of the OMI (for example, the CMD and SEQ columns for the {\tt
FNP} task), as predicted by the model.

\vspace*{-0.05in}
\subsection{NTD as a General Human}
\vspace*{-0.05in}

\mumble{Insert some nice blurb here about how the NTD actually
understands ``The man likes the horse,'' and, ``The apple is red,''
and when shoved into the laboratory, can exhibit some of the behavior
of NOVA.  Once we've actually shown this to be true.}

\section{Discussion}
\label{Discussion}

Although the integration of three independently developed cognitive
models was a non-trivial task, it proceeded more smoothly than
expected, thanks to the common Soar framework and some guidance from
Allen Newell's \citeyear{New90} model of immediate behavior.  This
model was dubbed PEACTIDM because of the set of stages it proposes --
Perception, Encoding, Attention, Comprehension, Tasking, Intention,
Decoding, and Motor.  These stages are both necessary and sufficient
to describe immediate interaction tasks, where a simple stimulus is
presented and a speeded response is expected.  Tasks such as pressing
a button in response to a light \cite{Luc86}\quest, responding when a
probe item is present in a set of items recently presented
\cite{Ste69}, and typing \cite{Joh88b} can be well explained at this
level.  Our use of this model takes it in a different direction, using
it as a metaphor and an additional constraint.  It suggests one
important consideration: that, in order for immediate behavior to
arise, the initial structure of the cognitive system must make it
possible to learn PEACTIDM-like operators.

Cognitive operators in Soar may only be applied sequentially, a
controversial aspect of the architecture that is motivated by several
different results.  A number of models developed for user interface
evaluation, including the Keystroke-Level Model \cite{CarMorNew80},
the Model Human Processor and GOMS \cite{CarMorNew83}, and CPM-GOMS
\cite{Joh88b,Joh90} have all had much success in prediction using
sequential cognitive operators. \mumble{Bonnie says ``Red Herring, I
think.  How does Newell argue the point in the UTC?''  See section
3.7, roughly pp. 133-135.  I find this argument totally unsatisfying
myself, which is why I didn't want to rely on it.  The strongest
argument in this section is the linear effect in Schneider \&
Shiffrin's varied mappings, and this only argues that bottlenecks
exist somewhere, not that all of cognition falls into the same pot.}
This sort of prediction requires a
correspondence to be drawn between cognitive operators and human
performance times.  Soar relies on this kind of correspondence,
predicting the time taken by humans to perform a particular task.  
%$$$$
For
each operator used in Soar, the effective duration, which should fall
in a range of about 30 to 300 milliseconds, is estimated independently
from the properties of the neural system and from the general
character of human behavior \cite{New90}.  In NTD-Soar, we have
assumed a fixed duration of 50 ms per operator, a value used
successfully in a number of other cognitive models
\cite{Joh88b,Wie92,Lew93b}.  By sharing these constraints with other
researchers working in a variety of domains, we hope to evaluate both
the sequential operator model and the 50 ms duration.
It is important to 
keep in mind that the computer need not necessarily evaluate or
``run'' the model in ``real-time'' in order to make useful
predictions.  The computer hardware and the implementation may leave
the evaluation slower than modelled behavior; in other cases, the
evaluation may actually happen faster than the behavior itself.  But
in either case, the model is making the same prediction about human
performance.

It is often suggested\grumble{refs...} that such sequential cognitive processing cannot
successfully model real-time human behavior.  Unlike some
faculty-based\mumble{I guess I need to expand this so that the
alternative is understood.} models, NTD-Soar cannot, for example, perform language
comprehension and initiate a motor action simultaneously.  However,
in memory retrieval (below the level of deliberate action in Soar),
information from different sources {\em is} available
in parallel.  Only when a variety of information is collected and a
single commitment made (the selection of an operator) does decision
making become serialized.  Furthermore, under conditions where humans
perform multiple high-level activities in parallel, it is hard to
distinguish an implementation by true parallelism from an
implementation that interleaves small components of the individual
activities.  NTD-Soar uses the interleaving model, doing so at the
operator level, and explaining the apparent parallelism by an ability
to switch fluidly and opportunistically among tasks. 
The advantage of the NTD-Soar model is that the operators remain
independent, and may be evoked in different combinations when
different tasks are selected, without the need to relearn how to
combine the behaviors.  Another Soar
model has been developed \cite{Cov92} which takes a different
approach, showing that operators which perform multiple behaviors in
parallel can be deliberately constructed within the serial framework.

Some unexpected issues arose in the integration process that led to
changes in the components.  The starkest changes were those made in
the NTD model itself.  Because the original model was constructed
without consideration for learning issues, and did not provide an
effective organization for the interleaving and interruption required
in the real-time model, much of the knowledge was reorganized.  The
character of the knowledge remained the same, but the occasions for
its application changed significantly.  Certain knowledge needed to be
cast in new forms, as exemplified by the discourse knowledge that is
now used to discriminate among options provided by language, rather
than setting up a task for language to accomplish.

The inclusion of the comprehension system in
a real-time domain brought out several new issues that did not arise
when NL-Soar was developed.  Because new words arrive asynchronously,
rather by the demand-based attention mechanism used in the original
NL-Soar model, processing on earlier words may be in almost any state
when the next word arrives.  The NL-Soar model posits that whenever a
word cannot be processed recognitionally, an attempt is made to learn
how to process that word.  Because the learning process is much longer
and more complex than the recognitional processing would be, new words
often arrive while learning is taking place.  In this circumstance, a
default behavior must be chosen -- either the learning may be allowed
to continue while the new word is ignored, or learning must be
interrupted to shift cognitive resources to the new word.  Of course,
this choice need not be made once and for all; for example, knowledge
could be applied while in the lower problem spaces to decide that in
this particular circumstance, it would be okay to abort on the arrival
of the next word even if that was not the default behavior.

Of course, a decision on this issue interacts with another important
decision, that of the grain size of the comprehension operators.
How much work should a top-space operator do?  There are several
constraints on this.  First, since every operator takes the same
amount of time, a larger grain size means more work gets accomplished
in the same period.  However, operators of a larger grain size require
more work to learn, and typically are applicable in fewer situations
because they are more specific (they exhibit less {\it transfer}
across situations).  This means that a lot more learning must go into
the development of such operators.  If we choose an interruptible
learning mechanism, then we may {\em never} be able to learn operators
that are more complex than some particular limit, because we will
always be interrupted before learning has completed.

To simplify matters, NL-Soar has already made a commitment on grain
size.  This decision was based partly on the need to accomplish a
certain amount of work in a certain fixed time: recognitional
processing allows at most 250 to 300 ms per word, so no more than five
or six cognitive operators.  On the other hand, the issue of transfer
pushes NL-Soar toward smaller operators.  The current model includes
two or three semantic and syntactic operators and a variable number of
referent-resolution operators.  The amount of work required to learn
even these simple component operators makes the learning process
fairly lengthy.  To avoid falling terribly behind in processing the
new input, the interruptible learning mechanism was chosen for the
new real-time version of NL-Soar.

This choice provides an additional constraint on language
comprehension not present in the original NL-Soar model.  The question
was, under what circumstances could NL-Soar still learn to comprehend
the utterances it heard.  The somewhat surprising answer was that,
even listening to the full-speed expert dialogs, the model was able to
learn to comprehend them, but that in some cases this learning might
require hearing the words {\em multiple times}.  A typical result from
Soar models that do not address the real-time issue is that they learn
fully recognitional performance in a single encounter with the input,
and from that point on exhibit the same behavior every time the input
is presented.  But because learning takes place in deeper subgoals
even if the highest subgoal doesn't get a chance to complete,
eventually the time required to complete the learning becomes short
enough that it can be accomplished before the next word arrives.

Changes to the NOVA model came mostly in the form of extensions,
rather than theoretical modifications.
To extend the NOVA model to {\it overt} visual attention, including
eye movements, we have taken the approach \cite<suggested by>{Rem80}
that eye movements are preceded by a shift of attention to a location
near the desired fixation point.  We allow covert attention shifts to
perceptual units that fall outside the fovea; at this point, enough
information becomes available to allow cognition to direct an eye
movement to that location.  Once the eye movement
completes, the attended unit will fall within the fovea, and so can be
resolved and processed by cognition.\footnote{We explored models that
involved errorful eye movements, but concluded that the additional
complexity did not add substantially to the usefulness of the model.}

Beyond the addition of eye movements to the NOVA model, a set of
cognitive operators compatible with NOVA's perceptual and attention
mechanisms had to be developed.  Much of the framework for this is
specified by NOVA, including types of operators and their functions,
which are specific subtypes of the attention and comprehension
operators mentioned above.  However, the conditions under which
particular operators are appropriate still needed to be specified for
the NTD domain.  To do this, the visual information requirements for
each of the tasks and subtasks specified by the domain reasoning model
were analyzed.  This produces a set of criteria for the proposal and
selection of visual scanning operators that depends on the character
of the currently selected task.  In this manner, the operators
available recognitionally may change as a function of the task
currently selected, another example of taskification.

In one sense, the Soar architecture is the only part of the
integration that remained completely unchanged.  Certainly the
cognitive mechanisms, states, operators, problem-spaces, impasses, and
so forth were a constant.  In another sense, however, Soar itself has
been extended by this research.  The addition of a phonological buffer
(now used in many systems interested in language comprehension), of
certain mechanisms for visual perception, and even of the strict,
lock-step simulation of real time to match cognitive time, have all
had an impact on the ways Soar is and can be used.

Soar's impact on the integration, and the constraints it brings, play
a critical role in shaping the NTD model.  Its most obvious
contribution was to make possible the integration of three
independently developed models, because of the assumptions they shared
about the character of cognitive behavior.  But Soar, along with the
analysis in {\it Unified Theories of Cognition}, also provides a model
for communication between capabilities, and mechanisms for
prioritizing them and selecting a single cognitive operator at any
point in time.  In addition, this constraint of serial cognitive
operators, each taking 50 ms, also provides a valuable means for
evaluating the model in comparison to human data.

The construction of the NASA Test Director model, initially intended
as a mechanism for user interface evaluation, has led to an in-depth
exploration of integrated cognitive models.  Because the NTD model
built upon several sophisticated models of particular capabilities,
the final model exhibits both the breadth of the domain and the depth
of these capabilities.  The combined model not only fits data in the
NASA domain, but retains the capacity to make valid predictions in
natural language comprehension and covert visual attention.  In
addition, extensions to these capabilities were shown to make
plausible predictions in new domains, including language learning and
overt visual attention.

The NTD model was developed in Soar, a cognitive architecture that
posits a serial bottleneck in cognition and provides a mechanism for
evaluating the durations of cognitive activities.  Although the model
does not yet cover the full range of NTD behaviors, it does exhibit
parallel motor activities (though not parallel {\em initiation} of
these motor activities), and shows cognitive processing times well
within the norms of human performance.  Thus, despite the hypothesis
(espoused by Fodor and others) that a serial framework cannot match
human performance, particularly in its apparent parallel nature, the
NTD model provides a demonstration of exactly such a match.

Finally, our exploration has led to a better understanding of what
is required to make independently constructed capabilities compatible,
and to some particular solutions for communication between the
combined capabilities.  In NTD-Soar, these elements include a shared
representation for shared information, explicit knowledge about
cross-task priorities, and mechanisms for learning and compiling
knowledge from different tasks together.

\section*{Acknowledgements}

We wish to thank Richard Lewis for extensive help and feedback during
the integration of NL-Soar, Robert Rubinoff, who is working on
generation within NL-Soar, and Roger Remington and Mark Wiesmeyer, who
provided many valuable insights into visual attention.
This research was supported in part by the NASA, Grant No. NAG 2-634
and in part by the Wright Laboratory, Aeronautical Systems Center, Air
Force Materiel Command, USAF, and the Advanced Research Projects
Agency under grant number F33615-93-1-1330.  The views and conclusions
contained in this document are those of the authors and should not be
interpreted as necessarily representing the official policies or
endorsements, either express or implied, of NASA, Wright Laboratory,
or the U.S.~Government.  The U.S.~Government is authorized to
reproduce and distribute reprints for Government purposes
notwithstanding any copyright notation thereon.

\vskip 0.2in
\bibliography{ntdnl}
\bibliographystyle{theapa}

\section{Cruftnstuff}

For example, one visual task frequently performed by the NTD (about
40\% of the tasks in one segment of the launch) is the
{\tt FIND-NEXT-PARTICIPATION} ({\tt FNP}) task.  Whenever this task is
selected, the NTD model searches the ``CMD'' and ``RESP'' columns of
the OMI for the word ``NTD'' to locate the next step involving the
NTD.  First, it makes sure it is looking at the correct columns, and
if not, moves its eyes there.  Next, it performs a series of attention
shifts, eye movements, and ``recognize'' operators (a type of
comprehension) to evaluate whether ``NTD'' has been found.  Recognize
operators are selected when an attended unit within the fovea has not
been evaluated; intrafoveal attention operators are selected when
there are units in the fovea that have not received attention yet; and
extrafoveal attention, followed by an eye movement, is performed when
there are no units within the fovea that still require processing.  If
this search fails to find anything before the end of the page, it may
turn to the next page.   It is important to note that this
particular procedure reflects only one way of doing the task, and
other NTDs or the same NTD at different times might use different
methods.  Each of these actions is conditioned on the selection of
the FNP task, as well as on knowledge about the status of the search
so far.  If at any point ``NTD'' is found or the task is changed, as
might happen when someone else summons the NTD, the set of operators
available will change to reflect possible relevant actions.  (This
second, external means of termination is not displayed in the figure.)

\begin{figure}[htbp]
\centerline{\psfig{figure=fnpflowchart.ps,height=8in}}
\caption{The algorithm for the {\tt FIND-NEXT-PARTICIPATION} task.}
\label{fig:fnpflowchart}
\end{figure}

\end{document}
