Multimodality and IF
Cristina Barbero, Emanuele Pianta, Massimo Zancanaro
20 March 2000

Comments by UKA (Tanja Schultz, Hagen Soltau, and Florian Metze)
May 18, 2000

1 What's a gesture?
In NESPOLE! a gesture can be one of the following actions:
•pointing: selecting an object on a whiteboard (or equivalent) using a pointing device (e.g. mouse, pen, finger on a touchscreen)
•selection: selecting among a finite number of items of a menu, a palette, a list etc.
•free-hand strokes on a whiteboard: the user is free to draw a gesture using a pointing device

Am I right in assuming that mouse gestures are only valid in conjunction with a click of a button, i.e. just moving the mouse on the picture of a hotel (without pressing a button) is not a valid gesture?
2 Analyzing gestures
While pointing and menu selections are simple gestures whose interpretation is not problematic, free-hand strokes are complex gestures that can be analyzed at different levels:
•morphological level: each stroke is interpreted as a kind of elementary graphical object (e.g. line, poly-line, circle, square ecc...)
•lexical level: sequences of strokes are segmented and interpreted as complex graphical objects (e.g. three lines as an arrow, two lines as a cross)
•semantic level: what is the meaning of a lexical element (e.g. a cross is interpreted as a delete command)

The morphological interpretation of free-hand strokes is provided for free by the whiteboard. Lexical interpretation is build on the top of morphological information and of course semantic interpretation is based on lexical analysis. 

One of the issue of the NESPOLE! scenario is whether we need to analyse  free-hand strokes at the semantic or at the lexical level.  In other words,  whether gestures  need to be translated or can simply be recognized and re-produced.  Moreover, even in the lexical approach, it may be the case that it will suffice to just segment a sequence of strokes without classifying it as a specific object (e.g. recognizing a sequence of three strokes as a single gesture without classifying it as an arrow). Let’s call this kind of analysis partial lexical analysis. 
It is worth noting that the lexical approach is based on the assumption that the gestures we are dealing with in the NESPOLE! scenario have the same meaning in different languages.

Lexical analysis seems to be a good basis for enhancing the capabilities of the system. In a first step, the mapping between lexical and semantical units could be straightforward, as long as we have a limited number of gestures. However, we believe that the format used needs to be flexibel enough to support semantical interpretation to compensate for different capabilities of the hardware used at the customer and agent site. For example, a whiteboard's green pen on an agent's site may need to be represented on a monochrome display of the customer site without the agent's active participation in the mapping.
3 Synchronizing gestures and verbal events
There are two ways of coordinating the production of verbal and gestures events:

•time-based anchoring: gestures are re-produced with a delay from the start of the translated sentence which is the same as the original sentence
•concept-based anchoring: the gesture is linked to a concept in the IF (using its unique id), the generator annotates the part of the generated sentence that represents the concept – the reproduction starts when the synthesizer reaches that annotation
4 Representing multimodal communicative acts
There at least two approaches to the problem of integrating the representations of gestures and verbal events. The first approach consists in building a unified representation of a multimodal communicative act, abstracting away from the verbal or gesture modality. Following this approach a unique representation could correspond to either a verbal event without any gesture, a verbal event with gestures, or a gesture without verbal events (see [Johnston et al.], [Vo and Woo]).
The second approach consists in building a two-folded representation describing separately the verbal and gesture modality with pointers from one modality to the other.
The first approach requires more theoretical and practical work and would be really beneficial or necessary only in a man-to-machine or cross-modality communication scenario. The second approach is simpler but still meets all the requirements of a speech-to-speech translation system. Thus, we propose that a gesture information channel  parallel to the verbal events channel is introduced in the NESPOLE! architecture. 

How could gestures and verbal events be connected? For example, it is conceivable that a gesture appears at another time than the related verbal event. Also, many gestures (e.g. circling) need a certain amount of time and could extend over more than one SDU. How could this information be represented? A negative time-stamp adds temporal flexibility to linking of gestures and verbal events but does not address the problem of finding the right alignment.
4.1 Gesture Descriptors 

Gestures are described by gesture descriptors, that is triples:

<gesture-id, time-stamp, gesture-description>

A gesture descriptor (GD) consists of a unique identifier, a reference to the time at which the gesture occurred (see below) and a description of the gesture.
All descriptions contain the type (either pointing, selection or freehand) and the context (see below). Other information depend on the value of  the type attribute. For pointing, we specify the object. For selection, we specify the selected item and its arguments, if any. For freehand gestures, the minimal information is the list of strokes (lexical analysis). In case of full lexical analysis the type of gesture (eg. cross, arrow, etc.) is specified, and in case of semantic analysis the meaning is specified as well. Here are some examples, one for each gesture type:

(gesture-id=G12, time-stamp=230304, 
 gesture=(type=selection, context=menu-file,
          item=(load, object=”pic1.jpg”))

where item is the item selected from a menu, in this case the action of loading an image (specified in the object attribute). 

(gesture-id=G25, time-stamp=257684, 
 gesture=(type=pointing, context=(button1, button2, button3),          
          object=button3))

in this case the context is the set of all the selectable objects on the speaker’s whiteboard when the pointing took place.

(gesture-id=G25, time-stamp=257684, 
 gesture=(type=freehand, context=”pict1.jpg”, 
          strokes=(line=(…), line=(…)), 
          lex=cross, sem=(delete, obj=<OBJ123>)))

the list of strokes is the minimal information for freehand gestures (how strokes are represented depends on the particular implementation of the whiteboard); in case of full lexical analysis, the lex attribute will be set and in case of semantic analysis the sem attribute will be set too.

Gesture descriptors are grouped in Gesture Descriptors Sequences (GDS). Each GDS is associated to some verbal event. More specifically, we propose that GDSs are associated to Semantic Dialogue Unitis - SDUs (alternatively, they could be associated to turns). In practice this means that each IF will have an associated (possibly empty) GDS. This approach requires of course that the sequence of gestures that goes with a turn be segmented by the analysis chain in GDSs, each corresponding to an SDU/IF.

In the rest of the document, two aspects are clarified, i.e. the nature of time stamps and the representation of the association between gestures and parts of the IF (concept anchoring). Both aspects are related to the issue of the synchronization between gestures and verbal events.
4.2 The nature of time stamps
We envisage two solutions. The time stamp can take as zero-time the starting of the turn or the starting of the SDU. The first solution is simpler than the second. Also, the first solution is feasible if we assume, as was the case in C-Star II, that the turn is the communication unit between the analysis and generation chains. However, if we intend to use time stamps as one of the means to achieve the synchronization between gestures and verbal events, adopting the start of the turn as zero-time could produce unacceptable results. Think of pauses, coughs, false startings, etc., that are contained in a turn on the speaker side but are never reproduced on the hearer side. Thus, we propose to take the starting of the SDU as zero-time. This will guarantee a more accurate synchronization between gestures and speech. 

If we accept this approach, we should consider the possibility that a gesture coming with an SDU is accomplished before the actual starting of the SDU. As a consequence we must allow time stamps to get negative value. For instance, time-stamp=-5 would mean that a gesture was accomplished 5 time-units before the starting of the associated verbal event (e.g. the speaker points to a picture and says "this").
Another technical point related with time stamps is the necessity for the gesture and speech channels to agree on a common reliable time gauge.
4.3 Representing Concept Anchoring
Time stamps at the SDU level guarantee a first simple way to resynchronize gestures and verbal events during the generation chain. However a more accurate synchronization can be obtained through concept anchoring, that is the association between gestures and specific fragments of an SDU. The most straightforward solution to the problem of representing concept anchoring is the use of pointers from a fragment of IF to one of the gestures contained in the GDS coming with the IF. For instance, suppose that the speaker says

"I'll travel to Paris in November"

and makes a circle in a shared map while saying "Paris". 

We can represent this communicative event as

IF: c:give-information+features+trip
    (destination=(paris, gesture=23), time=november)
GDS: ((gesture-id=23, time-stamp=83, 
       gesture=(type=freehand, context=”france-map.jpg”, strokes=(…))))

In the GDS,  the free-hand strokes have been interpreted at the partial lexical level and therefore the lex and sem attributes are missing.
As far as the IF syntax is concerned, we propose to add a 'gesture' attribute that (like the 'quantity' attribute) can appear within the value of any other attribute. The value of the gesture attribute must be the identifier of one of the gestures described in the associated GDS.
This solution has one limitation, due to the very nature of the IF as it is currently defined. Suppose that a gesture is accomplished in association to a verbal element that is represented in the IF by a concept but not an attribute. For example the speaker accomplishes the above communicative act, but associates the gesture to the verb "travel" instead of the locative "Paris". In this case, there would be no means of associating the gesture to the 'trip' concept, which roughly corresponds to the "travel" verb in the SDU. This limitation could be overcome by assuming that the gesture attribute included at the top level in the attributes list is associated to the main action of the event described by the IF. 

Sounds good to us.
References
[Johnston et al.] M. Johnston, P. R. Cohen, D. McGee, S. L. Oviatt, J. A. Pittman, I. Smith “Unification-based Multimodal Integration” in Proceedings of 35th Annual Meeting of the Association for Computational Linguistics, July 1997
[Vo and Woo] M. T. Vo and C. Wood, “Building and Application Framework for Speech and Pen Input Integration in Multimodal Learning Interfaces” in Proceedings of ICASSP96, 1996