Dealing with gestures in Nespole!
Massimo Zancanaro 
zancana@irst.itc.it
ITC-irst
April 4th, 2000

Gestures in Nespole!
In the context of Nespole! project, we will focus on deictic gestures on a whiteboard. The term deictic refers to the communicative function of referring to objects in the world (as opposed, for example, to emblematic gestures like the “thumb up” that signals “ok”). We will focus on the following three types of gestures: 
•	pointing: selecting an object using a pointing device (e.g. mouse, pen, finger on a touchscreen)
•	selection: selecting among a finite number of items of a menu, a palette, a list etc.
•	free-hand strokes: the user is free to draw a gesture using a pointing device
Technically speaking, only the third type can be called gestures, yet we will overload the meaning of the term in order to have a more usable scenario.
 A scenario
Both the agent and the client can access a whiteboard while speaking. The agent can (i) load images (like maps, timetables, etc.) by means of selection gestures (ii) load buttons or define hotspots (by means of selection gestures) that the client can then select using pointing gestures. Both the agent and the client can draw free-hand strokes to make reference to objects in the map (generally, on the image loaded by the agent).
•	Example 1: the client wishes to plan a trip to Trento, the agent loads the map of north-east Italy (selection gesture) and says “you will arrive at Verona airport” circling an area on the image that correspond to Verona (free-hand gestures). “If you rent a car, take Highway A31” drawing a line from Verona to Trento along the path of the highway (free hand gestures).
•	Example 2: the client wishes to make a one-day trip to Venice during her holidays in Trentino. The agent loads on the whiteboard a train timetable (selection gestures). The agent says: “you can leave at 8 am from Trento to arrive at Venice S.Lucia at 10 am” selecting with the pen the corresponding row on the timetable: since the timetable rows has been previously defined as hotspots, selection causes the row to be highlighted (pointing gesture). The client replies “I rather prefer to take the train at 10.30” pointing to the corresponding row to highlight it.
Interpreting gestures
The extent to which the system should be able to interpret the meaning of gestures is not clear yet. The minimal functionality is to simply share the whiteboard without any attempt to synchronise gestures with the translated speech. A further step may be to construct some sort of alignment between the two modes without a full ‘semantic’ interpretation of gestures. 
While pointing and menu selections are simple gestures whose interpretation is not problematic, free-hand strokes are complex gestures that can be analysed at different levels:

•	morphological level: each stroke is interpreted as a kind of elementary graphical object (e.g. line, poly-line, circle, square ecc...)
•	lexical level: sequences of strokes are segmented and interpreted as complex graphical objects (e.g. three lines as an arrow, two lines as a cross)
•	semantic level: finding out the meaning of a lexical element (e.g. a cross is interpreted as a delete command)

In a previous report (Barbero, Pianta, Zancanaro, 2000), we suggested to interpret free-hand strokes up to a partial lexical analysis, that is segmenting a sequence of strokes without classifying it as a specific object (e.g. recognising a sequence of three strokes as a single gesture without classifying it as an arrow). The gestures will then be anchored to an IF and finally reproduced in the translated utterance. We argued that there are two ways of anchoring a gesture to an IF:

•	time-based anchoring: gestures are re-produced with a delay from the start of the translated sentence which is the same as the original sentence
•	concept-based anchoring: the gesture is linked to a concept in the IF (using its unique id), the generator annotates the part of the generated sentence corresponding to the concept – the reproduction starts when the synthesiser reaches that annotation.

Further details on concept-based anchoring are discussed in the internal report (Barbero et al. 2000).
Tentative design of the experiments
The main goal of the experiments is to compare concept-based anchoring to time-based anchoring with respect to a human alignment of gestures on the translated speech. 
The experiments will be organised in the following phases:
1.	collection of a set of monolingual multimodal dialogues and (semi-)automatically production of the corresponding IF
2.	translation of the dialogues (manually or semi-automatically) and manual alignment of  gestures (this is the set B, the baseline)
3.	manual anchoring of gestures on IF (concept-based anchoring) and (semi-)automatic generation of the set C
4.	generation of set T using time-based anchoring 
5.	computing the relative errors of concept-based (set C) and time-based (set T) with respect to manual alignment (set B). 
The null hypothesis of the experiment is that concept-based anchoring is not better than time-based. 

OTHER GOALS?

In order to manually annotate dialogues, a multimedia annotation tool has to be used. One possibility is to use the MITRE multi-modal logger, available (without charge to use for internal purposes only)
 at http://www.mitre.org/technology/logger/, another possibility is to augment the tool used for manually annotate IF (does it exists?)
Whiteboard functionalities
The whiteboard will support the three types of gestures described above. It will be able to perform a partial lexical analysis on free-hand strokes (that is, to parse each stroke has an elementary object and to group the strokes into a complex graphical object). For each gesture, an object will be built (see below) and sent to the server. The whiteboard will also be able to receive objects in the same format and reproduce timely the gestures. Figure 1. depicts a tentative class diagram for the main classes of the whiteboard.

List of functionalities:

Code	Functionality	Description	Comment	
Load-Im	load image	load an image on the whiteboard, only one image can be present on the whiteboard at a time. That image represents the context for free-hand strokes.
An object of type select is built and sent to the server.	agent side only	
Load-HIm	load image with hotspots	load an images with hotspots that will be the context for further  pointing  (or free-hand) gestures.
An object of type select is built and sent to the server.	agent side only	
M-Point	set the “pointing” mode	when this mode is set, the device is used for pointing rather than for drawing strokes.
It is allowed only in the context of an image with hotspots. 		
M-FreeHand	set the “freehand” mode	when this mode is set, the device is used for drawing strokes. It is the default mode when the context is not null.		
Stroke	draw a stroke 	a stroke is drawn on the whiteboard and the corresponding stroke object is built. The strokes are then built into gestures and sent to the server. 		
Point	point on a hotspot	the hostspot is activated and the corresponding object is built and sent to the server		
Time	set the starting time for the time counter	It is used to synchronise the whiteboard with speech recogniser		
Perform-gst	perform a give gesture object	gesture objects are sent by the server (which takes care of properly synchronise them with the speech). The whiteboard should perform a gesture as soon as it receives the corresponding gesture object.		
 
References
C. Barbero, E. Pianta, M. Zancanaro, Multimodality and IF, Nespole! Internal Report #6.1.1, March 2000