05-830, User Interface Software, Spring 2001
Homework 1: Benchmark Description
Data Representation Dialogue Tutor
Ryan Shaun Baker
http://www.hcii.cmu.edu/~rsbaker/benchmark.html
Introduction
The benchmark is an interface for a dialogue (“3rd-generation” is the more commonly used terminology) tutor which teaches data representation. It is modeled after those used in the PACT Center’s high school and middle school cognitive tutors – though with some differences owing to the difference in intended task -- and is intended for use in by middle school students. It gives students a question and a data set, and then asks the student to choose a style of representation and draw that representation for the given data set, and finally asks them to answer the question based upon that drawn representation. If the student makes an inappropriate choice at any point, it engages in a “knowledge-construction dialogue” with them, asking them subsidiary questions to the one they missed, to arrive at the answer. (The dialogue tree used for the benchmark will be considerably simpler than that used in a real tutor, for reasons of simplicity) The student interacts with the tutor through answering directly-given questions and through creating the representation, actually drawing a graphic using the equivalent of a pencil tool. Several menu commands will also be available for functions such as exiting, looking at metrics of performance progress, and going on to the next problem.
This is representative primarily of two styles of interfaces, although it has aspects of one other.
- Question-Answer interfaces, through the dialogue between the student and the tutor; including dialogue trees
- Direct Manipulation, through having students draw a graphic using the equivalent of a pencil tool.
- Menus are also used, to govern program operations outside of the dialogue
This benchmark is a good benchmark for these styles. One reason is because it is an authentic task which combines Q&A with Direct Manipulation. It captures the most salient feature of Question-Answer systems, the process of the computer asking questions which the user answers; furthermore, the con that a user cannot correct previous errors or change past input is no longer a negative feature; mistakes are seen in this context as something to provide tailored assistance for, and the dialogue tree returns to the previous point and continues after helping the student fix their mistake. It captures fewer of the aspects of direct manipulation systems. While it does have the central fact that the student actually constructs their graph by clicking and dragging on the screen as if they were using a pencil on paper, including the visual model aspect and the fact that the results of actions are immediately reflected onscreen, it lacks the “object-oriented” aspects of most direct manipulation systems. There are no objects which can be manipulated by clicking on them.
Description
There will be a window where the dialogue between the student and the tutor takes place. It will include a box where the tutor makes comments and gives questions to the student, a box where the student inputs their answers, a button allowing the student to enter their comment, and a button allowing the student to say “Huh?” in response to the tutor. (which asks for a hint or a further hint). All buttons and fields will be disabled (grayed out for buttons) while the student draws the graph. It will also include (or have near it) some sort of picture of the tutor. This picture will be animated to have expressions based on student performance – smiling and wagging its tail when the student gives a correct answer, looking at papers at its desk when the student goes “Huh?”, crossing its arms and smiling when the student finishes a whole problem, perking its ears up when the student gets an answer wrong in a known way, and looking confused and stroking its chin when the student gives an unknown answer.
See figure 1…
Figure 1
There will be a second window which the user does the representation drawing on.
Its contents will be modified both by the user drawing in it, and by the tutor adding things
into it (axis labels, scale) in response to student answer to its questions. The student enters their
drawing by clicking a “Grade” button at the bottom of the window (not included in figure 2). This
button will be grayed out at all times other than when the student is expected to draw the graph.
Determining whether the student has drawn the correct graphic will be based upon:
The presence of five vertical lines in the correct places (within 25 pixels) for each box and whisker.
A vertical line will be defined as a point in more than 25 of 30 pts vertical, with no more than ten pixels spread.
See figure 2 for what this will look like…
Figure 2
There will be a third window which contains the data set under consideration. It will contain a simple table of the data, as well as the overall question which we are trying to answer.
Finally, there will be a menu which has the other operations which students can do at any time.
This includes commands to
- Exit the program
- Go to the next problem (starts the below dialogue again at (1))
Dialogue Tree
For the purposes of this benchmark, a very simplified dialogue tree will be used which demonstrates the possibility of dynamic dialogue without using the cognitive modeling necessary to actually choose which dialogue path to take, or generating those paths on the fly, as an actual tutor with this interface would do. Only one data set and dialogue tree will be used. Parsing will take place by simple keyword searching on the student’s input. Mis-spellings will not be accounted for, for the purpose of this benchmark. Note that Huh?,Unknown, and the various wrong answers may transit together, but will still cause different behaviors on the part of the agent, as discussed above.
The paths below are only as much as is necessary to expose all of the interface features. No attempt to actually model the problem-solving process takes place.
The following transition table represents the dialogue:
|
State |
Inputs |
New State |
Correct? |
Interface Actions (not including agent) |
|
1 |
Huh?, Unknown, “Histogram”, “Scatterplot”, “Stem and Leaf Plot” |
2 |
N |
|
|
1 |
“Box Plot” |
3 |
Y |
|
|
2 |
“Price” AND “Regular or Natural”, “Price” AND “Regular/Natural” |
4 |
Y |
|
|
2 |
“Regular or Natural”, “Regular/Natural” |
5 |
N |
|
|
2 |
“Price” |
6 |
N |
|
|
2 |
Huh?, Unknown |
7 |
N |
|
|
3 |
Automatic Transition |
10 |
N/A |
|
|
4 |
Automatic Transition |
1 |
N/A |
|
|
5 |
“Price” |
4 |
Y |
|
|
5 |
“Regular or Natural”, “Regular/Natural”, Huh?, Unknown |
8 |
N |
|
|
6 |
“Regular or Natural”, “Regular/Natural” |
4 |
Y |
|
|
6 |
“Price”, Huh?, Unknown |
8 |
N |
|
|
7 |
Automatic Transition |
9 |
N/A |
|
|
8 |
Automatic Transition |
9 |
N/A |
|
|
9 |
Automatic Transition |
2 |
N/A |
|
|
10 |
Any input |
11 |
Y |
Interface draws the input as the graph’s x axis |
|
11 |
Any input |
12 |
Y |
Interface draws the input as the graph’s y axis |
|
12 |
Correct graph |
13 |
Y |
|
|
12 |
Median wrong |
16 |
N |
Area where median line should be is cleared |
|
12 |
25th or 75th percentile wrong |
15 |
N |
Area where 25th and 75th lines should be is cleared |
|
12 |
Min or max wrong |
14 |
N |
Area where min and max lines should be is cleared |
|
13 |
No transitions |
|
|
|
|
14 |
Automatic Transition |
12 |
N/A |
|
|
15 |
Automatic Transition |
12 |
N/A |
|
|
16 |
Automatic Transition |
12 |
N/A |
|
Tree:
