What is BNT-SM?

Bayes Net Toolbox for Student Modeling (BNT-SM) is an effort to facilitate the use of dynamic Bayes nets in the student modeling community. Dynamic Bayes Nets (DBNs) provide a powerful way to represent and reason about uncertainty in time series data, and are therefore well-suited to model a student's changing knowledge state during skill acquisition. Many general-purpose Bayes net packages have been implemented and distributed; however, constructing DBNs often involves complicated coding effort. To address this issue, we extend the popular Bayes Net Toolbox (Murphy, 1998) and tailor BNT-SM for the student modeling community.

BNT-SM inputs a data set and a compact XML specification of a Bayes net model hypothesized by a researcher to describe causal relationships among student knowledge and observed behavior. BNT-SM generates and executes the code to train and test the model using the Bayes Net Toolbox. BNT-SM allows researchers to easily explore different hypothesis with respect to the knowledge representation in a student model. For example, by varying the graphical structure of a Bayesian network, we examined how tutoring intervention can affect students' knowledge state - whether the intervention is likely to scaffold or to help students to learn.

How to download and use BNT-SM?

BNT-SM is implemented in Matlab, so you need to have Matlab installed and running. Commercial and student license of Matlab can be purchased from Mathworks. By the way, for those of you who are not familiar with Matlab, Mark S. Gockenbach has an excellent, introductory tutorial to Matlab.

Before we download BNT-SM, we like to thank Kevin Murphy for his kindness in distributing Bayes Net Toolkit (BNT), which BNT-SM based and heavily depended on. For those of you who are proficient in coding and would like to go to the low level BNT code, BNT can be downloaded from Source Forge. Kevin Murphy also has a nice tutorial to BNT and Bayes nets in general.

BNT-SM can be downloaded from here.

Now, with BNT-SM downloaded and extracted, launch Matlab and do

>> cd src
>> setup
>> cd ../model/kt
>> [property evidence hash_bnet] = RunBnet('property.xml');

property.xml is an XML file that specifies the Bayes net we are constructing.
In the directory, BNT-SM/model, you can find some other sample Bayes net specification and a small test set to get started.

The RunBnet function first constructs the Bayes net specified in the property_xml file and trains the Bays nets using the training data set specified in the property_xml file. Then, it will extract model parameters and output to the param_table specified in property_xml. Finally, the RunBnet function will perform inference on hidden nodes and output to the inference_result specified in property_xml.

A Walk-through Example of modeling Knowledge Tracing with BNT-SM

Knowledge Tracing (KT) [2] is an established technique for student modeling and was first used in the ACT Programming Languages Tutor [2]. The goal of KT is to estimate the student’s knowledge from his or her observed actions. At each successive opportunity to apply a skill, KT updates its estimated probability that the student knows the skill, based on the skill-specific learning and performance parameters and the observed student performance (evidence). Reye [6] showed that KT is special case of a DBN which assumes parameters do not change across time slices. More specifically, the conditional independence graph of KT can be drawn as Figure 2. (We rename the original KT variables L0 and t as already know and learn for consistent naming with other sections.) Knowledge Tracing

In the following sections, we will briefly go over the format of the input and output files. There are additional details in the file README.txt at the top level directory in the package.

property_xml

<?xml version="1.0"?>

<property>

        <input>
                <evidence_train>evidence.train.xls</evidence_train>
                <evidence_test>evidence.test.xls</evidence_test>
        </input>

        <inference>fast</inference>

        <output>
                <param_table>param_table.xls</param_table>
                <inference_result>inference_result.xls</inference_result>
                <inference_result_header>inference_result_header.xls</inference_result_header>
                <inference_is_prior>yes</inference_is_prior>
                <log>log.txt</log>
        </output>

        <structure>
                <var>F = 1; T = 2;</var>

                <nodes>
                        <node>
                                <id>1</id>
                                <name>knowledge</name>
                                <type>discrete</type>
                                <values>2</values>
                                <latent>yes</latent>
                                <field>knowledge</field>
                                <within>
                                        <transition>asr_accept</transition>
                                </within>
                                <between>
                                        <transition>knowledge</transition>
                                </between>
                        </node>

                        <node>
                                <id>2</id>
                                <name>asr_accept</name>
                                <type>discrete</type>
                                <values>2</values>
                                <latent>no</latent>
                                <field>asr_accept</field>
                                <within></within>
                                <between></between>
                        </node>
                </nodes>

                <eclasses>
                        <eclass>
                                <id>1</id>
                                <formula>P1(knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P1(T)</eq><init>0.68</init><param>L0</param>
                                        <eq>P1(F)</eq><init>1-P1(T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P1_dir(T)</eq><init>6</init>
                                        <eq>P1_dir(F)</eq><init>9</init>
                                </dirichlet>
                        </eclass>

                        <eclass>
                                <id>2</id>
                                <formula>P2(asr_accept|knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P2(T|F)</eq><init>0.64</init><param>guess</param>
                                        <eq>P2(F|T)</eq><init>0.07</init><param>slip</param>
                                        <eq>P2(F|F)</eq><init>1-P2(T|F)</init><param>null</param>
                                        <eq>P2(T|T)</eq><init>1-P2(F|T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P2_dir(T|F)</eq><init>19</init>
                                        <eq>P2_dir(F|T)</eq><init>1</init>
                                        <eq>P2_dir(F|F)</eq><init>9</init>
                                        <eq>P2_dir(T|T)</eq><init>15</init>
                                </dirichlet>
                        </eclass>

                        <eclass>
                                <id>3</id>
                                <formula>P3(knowledge|knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P3(T|F)</eq><init>0.14</init><param>t</param>
                                        <eq>P3(F|T)</eq><init>0.00</init><param>forget</param>
                                        <eq>P3(F|F)</eq><init>1-P3(T|F)</init><param>null</param>
                                        <eq>P3(T|T)</eq><init>1-P3(F|T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P3_dir(T|F)</eq><init>2</init>
                                        <eq>P3_dir(F|T)</eq><init>0</init>
                                        <eq>P3_dir(F|F)</eq><init>9</init>
                                        <eq>P3_dir(T|T)</eq><init>0</init>
                                </dirichlet>
                        </eclass>
                </eclasses>
        </structure>
</property>

evidence.xls

Tab delimited text file. This is very important: it is not an Excel file
Two fields are essential: user and skill
BNT-SM will train a Bayes net for each skill, lumping data from all users. That is, BNT-SM train skill-specific models.
Evidence data should be comprehensive. That is, there shouldn't be missing data point.
Hidden variables and missing observations should be marked with NULL
For discrete variables, the values cannot be 0 because Matlab use it as array subscript (starting with 1). Therefore, we often increment the discrete variables by 1. In the case of a binary variable, 1 would be used for 0 or false values, wheres 2 would be used for 1 or true values.

user

machine_name

utterance_start_time

utterance_sms

target_word_number

skill

help

knowledge

correct

transcript_key

trn_correct

asr_accept

confidence_score

asr_confidence

fBS7-7-1990-02-03

LISTEN01-308-04

2000-05-12 17:52:27

640

WORLD

NULL

0.0668763

fDL7-5-1993-11-28

LISTEN01-334-04

2004-10-13 13:56:49

421

WORLD

NULL

0.0430581

param_table.xls

Since BNT-SM estimates skill-specific models, a Bayes net is trained for each skill in the training dataset.
BNT-SM outputs parameters that are specifified in the property_xml file

skill	num_users	num_cases	ll	L1	guess	slip	t	forget
skill_HELLO	14	23	-3.609297	0.744855	0.721432	0.000005	0.982517	0.000001
skill_WORLD	46	218	-90.177505	0.695366	0.634124	0.113612	0.256071	0.000001

inference_result.xls

The format of the inference_result.xls is identical to that of evidence.xls, except that BNT-SM performs inferences on the hidden variables and estimate their values.
For binary hidden variable X, the estimated value will be probability of X = 1 (represented X == 2 in Matlab)
For discrete hidden variables Y with values greater than 2, the probability of all values will be output in the form of [p1; p2; ... pn;]
By default, BNT-SM inferences posterior probability (after observing the evidence
If instead, you want to inference prior probability (before observing the evidence. For instance, in classic Knowledge Tracing), you can make a switch in RunBnet.m when calling inference_bnet.m

user

machine_name

utterance_start_time

utterance_sms

target_word_number

skill

help

knowledge

correct

transcript_key

trn_correct

asr_accept

confidence_score

asr_confidence

fCA8-5-1994-06-27

LISTEN01-302-04

2004-11-09 11:25:24

218

HELLO

0.801846

NULL

0.124751

fCA7-5-1994-06-27

LISTEN01-315-04

2005-01-24 11:33:12

468

HELLO

0.997498

NULL

0.0785152

Additional Output

For efficiency/debug purposes, we have also output the following files for you convenience:

log.txt - Helps finding out which skill the model is training/running on. The name of this file is definable in the property.xml file. Matlab does not update the screen often when running intensively on a large data set.
hash_bnet.mat - an object that stores all the trained Bnets in Matlab binaries.
evidence.mat - an object that stores the processed data set in Matlab binaries. Notice, as a preprocessing step, BNT-SM first constructs a structure out of the tab-delimited evidence file. When the data gets large, it can take quite a while for Matlab to load. Therefore, the loaded structure is saved as Matlab binaries for fast loading.

Project Listen

Training a Student Model to be used in the Reading Tutor

Specify the Bayes nets in the property_xml file.
Provide the data.
Execute RunBnet.m
Retrieve the model parameters from the param_table.
Derive the equation for student models.
Inference pknow and pcorrect using the equations derived by hand.
Verify the correctness of your derivation by comparing the pknow and pcorrect you derived to what BNT-SM estimated in inference_result.

Training the Out of Vocabulary (OOV) model

Make the observation that this task can be done by pre-processing the evidence.xls file
That is, sort the evidence.xls file by number of cases in each skill.
Then, for those skills that have number of cases below certain threshold (for example, less than 10 cases or 5 students), replace the skill name with "OOV"
Now the skill "OOV" estimated by BNT-SM is your OOV model