A Walk-through Example of modeling Knowledge Tracing with BNT-SM

Knowledge Tracing (KT) is an established technique for student modeling and was first used in the ACT Programming Languages Tutor. The goal of KT is to estimate the student knowledge from his or her observed actions. At each successive opportunity to apply a skill, KT updates its estimated probability that the student knows the skill, based on the skill-specific learning and performance parameters and the observed student performance (evidence). Reye showed that KT is special case of a DBN which assumes parameters do not change across time slices. More specifically, the conditional independence graph of KT can be drawn as the following Figure. (We rename the original KT variables L0 and t as already know and learn for consistent naming with other sections.)

In the following sections, we will briefly go over the format of the input and output files. There are additional details in the file README.txt at the top level directory in the package.

property_xml

<?xml version="1.0"?>

<property>

        <input>
                <evidence_train>evidence.train.xls</evidence_train>
                <evidence_test>evidence.test.xls</evidence_test>
        </input>

        <inference>fast</inference>

        <output>
                <param_table>param_table.xls</param_table>
                <inference_result>inference_result.xls</inference_result>
                <inference_result_header>inference_result_header.xls</inference_result_header>
                <inference_is_prior>yes</inference_is_prior>
                <log>log.txt</log>
        </output>

        <structure>
                <var>F = 1; T = 2;</var>

                <nodes>
                        <node>
                                <id>1</id>
                                <name>knowledge</name>
                                <type>discrete</type>
                                <values>2</values>
                                <latent>yes</latent>
                                <field>knowledge</field>
                                <within>
                                        <transition>asr_accept</transition>
                                </within>
                                <between>
                                        <transition>knowledge</transition>
                                </between>
                        </node>

                        <node>
                                <id>2</id>
                                <name>asr_accept</name>
                                <type>discrete</type>
                                <values>2</values>
                                <latent>no</latent>
                                <field>asr_accept</field>
                                <within></within>
                                <between></between>
                        </node>
                </nodes>

                <eclasses>
                        <eclass>
                                <id>1</id>
                                <formula>P1(knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P1(T)</eq><init>0.68</init><param>L0</param>
                                        <eq>P1(F)</eq><init>1-P1(T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P1_dir(T)</eq><init>6</init>
                                        <eq>P1_dir(F)</eq><init>9</init>
                                </dirichlet>
                        </eclass>

                        <eclass>
                                <id>2</id>
                                <formula>P2(asr_accept|knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P2(T|F)</eq><init>0.64</init><param>guess</param>
                                        <eq>P2(F|T)</eq><init>0.07</init><param>slip</param>
                                        <eq>P2(F|F)</eq><init>1-P2(T|F)</init><param>null</param>
                                        <eq>P2(T|T)</eq><init>1-P2(F|T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P2_dir(T|F)</eq><init>19</init>
                                        <eq>P2_dir(F|T)</eq><init>1</init>
                                        <eq>P2_dir(F|F)</eq><init>9</init>
                                        <eq>P2_dir(T|T)</eq><init>15</init>
                                </dirichlet>
                        </eclass>

                        <eclass>
                                <id>3</id>
                                <formula>P3(knowledge|knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P3(T|F)</eq><init>0.14</init><param>t</param>
                                        <eq>P3(F|T)</eq><init>0.00</init><param>forget</param>
                                        <eq>P3(F|F)</eq><init>1-P3(T|F)</init><param>null</param>
                                        <eq>P3(T|T)</eq><init>1-P3(F|T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P3_dir(T|F)</eq><init>2</init>
                                        <eq>P3_dir(F|T)</eq><init>0</init>
                                        <eq>P3_dir(F|F)</eq><init>9</init>
                                        <eq>P3_dir(T|T)</eq><init>0</init>
                                </dirichlet>
                        </eclass>
                </eclasses>
        </structure>
</property>

evidence.xls

Tab delimited text file. This is very important: it is not an Excel file
Two fields are essential: user and skill
BNT-SM will train a Bayes net for each skill, lumping data from all users. That is, BNT-SM train skill-specific models.
Evidence data should be comprehensive. That is, there shouldn't be missing data point.
Hidden variables and missing observations should be marked with NULL
For discrete variables, the values cannot be 0 because Matlab use it as array subscript (starting with 1). Therefore, we often increment the discrete variables by 1. In the case of a binary variable, 1 would be used for 0 or false values, wheres 2 would be used for 1 or true values.

user

machine_name

utterance_start_time

utterance_sms

target_word_number

skill

help

knowledge

correct

transcript_key

trn_correct

asr_accept

confidence_score

asr_confidence

fBS7-7-1990-02-03

LISTEN01-308-04

2000-05-12 17:52:27

640

WORLD

NULL

0.0668763

fDL7-5-1993-11-28

LISTEN01-334-04

2004-10-13 13:56:49

421

WORLD

NULL

0.0430581

param_table.xls

Since BNT-SM estimates skill-specific models, a Bayes net is trained for each skill in the training dataset.
BNT-SM outputs parameters that are specifified in the property_xml file

skill	num_users	num_cases	ll	L1	guess	slip	t	forget
skill_HELLO	14	23	-3.609297	0.744855	0.721432	0.000005	0.982517	0.000001
skill_WORLD	46	218	-90.177505	0.695366	0.634124	0.113612	0.256071	0.000001

inference_result.xls

The format of the inference_result.xls is identical to that of evidence.xls, except that BNT-SM performs inferences on the hidden variables and estimate their values.
For binary hidden variable X, the estimated value will be probability of X = 1 (represented X == 2 in Matlab)
For discrete hidden variables Y with values greater than 2, the probability of all values will be output in the form of [p1; p2; ... pn;]
By default, BNT-SM inferences posterior probability (after observing the evidence
If instead, you want to inference prior probability (before observing the evidence. For instance, in classic Knowledge Tracing), you can make a switch in RunBnet.m when calling inference_bnet.m

user

machine_name

utterance_start_time

utterance_sms

target_word_number

skill

help

knowledge

correct

transcript_key

trn_correct

asr_accept

confidence_score

asr_confidence

fCA8-5-1994-06-27

LISTEN01-302-04

2004-11-09 11:25:24

218

HELLO

0.801846

NULL

0.124751

fCA7-5-1994-06-27

LISTEN01-315-04

2005-01-24 11:33:12

468

HELLO

0.997498

NULL

0.0785152

Additional Output

For efficiency/debug purposes, we have also output the following files for you convenience:

log.txt - Helps finding out which skill the model is training/running on. The name of this file is definable in the property.xml file. Matlab does not update the screen often when running intensively on a large data set.
hash_bnet.mat - an object that stores all the trained Bnets in Matlab binaries.
evidence.mat - an object that stores the processed data set in Matlab binaries. Notice, as a preprocessing step, BNT-SM first constructs a structure out of the tab-delimited evidence file. When the data gets large, it can take quite a while for Matlab to load. Therefore, the loaded structure is saved as Matlab binaries for fast loading.

Project Listen

Training a Student Model to be used in the Reading Tutor

Specify the Bayes nets in the property_xml file.
Provide the data.
Execute RunBnet.m
Retrieve the model parameters from the param_table.
Derive the equation for student models.
Inference pknow and pcorrect using the equations derived by hand.
Verify the correctness of your derivation by comparing the pknow and pcorrect you derived to what BNT-SM estimated in inference_result.

Training the Out of Vocabulary (OOV) model

Make the observation that this task can be done by pre-processing the evidence.xls file
That is, sort the evidence.xls file by number of cases in each skill.
Then, for those skills that have number of cases below certain threshold (for example, less than 10 cases or 5 students), replace the skill name with "OOV"
Now the skill "OOV" estimated by BNT-SM is your OOV model