A Walk-through Example of modeling Knowledge Tracing with BNT-SM

Knowledge Tracing (KT) is an established technique for student modeling and was first used in the ACT Programming Languages Tutor. The goal of KT is to estimate the student knowledge from his or her observed actions. At each successive opportunity to apply a skill, KT updates its estimated probability that the student knows the skill, based on the skill-specific learning and performance parameters and the observed student performance (evidence). Reye showed that KT is special case of a DBN which assumes parameters do not change across time slices. More specifically, the conditional independence graph of KT can be drawn as the following Figure. (We rename the original KT variables L0 and t as already know and learn for consistent naming with other sections.)

Knowledge Tracing

In the following sections, we will briefly go over the format of the input and output files. There are additional details in the file README.txt at the top level directory in the package.

property_xml

<?xml version="1.0"?>

<property>

        <input>
                <evidence_train>evidence.train.xls</evidence_train>
                <evidence_test>evidence.test.xls</evidence_test>
        </input>

        <inference>fast</inference>

        <output>
                <param_table>param_table.xls</param_table>
                <inference_result>inference_result.xls</inference_result>
                <inference_result_header>inference_result_header.xls</inference_result_header>
                <inference_is_prior>yes</inference_is_prior>
                <log>log.txt</log>
        </output>

        <structure>
                <var>F = 1; T = 2;</var>

                <nodes>
                        <node>
                                <id>1</id>
                                <name>knowledge</name>
                                <type>discrete</type>
                                <values>2</values>
                                <latent>yes</latent>
                                <field>knowledge</field>
                                <within>
                                        <transition>asr_accept</transition>
                                </within>
                                <between>
                                        <transition>knowledge</transition>
                                </between>
                        </node>

                        <node>
                                <id>2</id>
                                <name>asr_accept</name>
                                <type>discrete</type>
                                <values>2</values>
                                <latent>no</latent>
                                <field>asr_accept</field>
                                <within></within>
                                <between></between>
                        </node>
                </nodes>

                <eclasses>
                        <eclass>
                                <id>1</id>
                                <formula>P1(knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P1(T)</eq><init>0.68</init><param>L0</param>
                                        <eq>P1(F)</eq><init>1-P1(T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P1_dir(T)</eq><init>6</init>
                                        <eq>P1_dir(F)</eq><init>9</init>
                                </dirichlet>
                        </eclass>

                        <eclass>
                                <id>2</id>
                                <formula>P2(asr_accept|knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P2(T|F)</eq><init>0.64</init><param>guess</param>
                                        <eq>P2(F|T)</eq><init>0.07</init><param>slip</param>
                                        <eq>P2(F|F)</eq><init>1-P2(T|F)</init><param>null</param>
                                        <eq>P2(T|T)</eq><init>1-P2(F|T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P2_dir(T|F)</eq><init>19</init>
                                        <eq>P2_dir(F|T)</eq><init>1</init>
                                        <eq>P2_dir(F|F)</eq><init>9</init>
                                        <eq>P2_dir(T|T)</eq><init>15</init>
                                </dirichlet>
                        </eclass>

                        <eclass>
                                <id>3</id>
                                <formula>P3(knowledge|knowledge)</formula>
                                <type>discrete</type>
                                <clamp>no</clamp>
                                <cpd>
                                        <eq>P3(T|F)</eq><init>0.14</init><param>t</param>
                                        <eq>P3(F|T)</eq><init>0.00</init><param>forget</param>
                                        <eq>P3(F|F)</eq><init>1-P3(T|F)</init><param>null</param>
                                        <eq>P3(T|T)</eq><init>1-P3(F|T)</init><param>null</param>
                                </cpd>
                                <dirichlet>
                                        <eq>P3_dir(T|F)</eq><init>2</init>
                                        <eq>P3_dir(F|T)</eq><init>0</init>
                                        <eq>P3_dir(F|F)</eq><init>9</init>
                                        <eq>P3_dir(T|T)</eq><init>0</init>
                                </dirichlet>
                        </eclass>
                </eclasses>
        </structure>
</property>

evidence.xls

usermachine_nameutterance_start_timeutterance_smstarget_word_numberskillhelpknowledgecorrecttranscript_keytrn_correctasr_acceptconfidence_scoreasr_confidence
fBS7-7-1990-02-03LISTEN01-308-042000-05-12 17:52:276408WORLD1NULLNULLNULLNULL20.06687631
fDL7-5-1993-11-28LISTEN01-334-042004-10-13 13:56:4942113WORLD2NULLNULLNULLNULL20.04305811

param_table.xls

skillnum_usersnum_casesllL1guesssliptforget
skill_HELLO1423-3.6092970.7448550.7214320.0000050.9825170.000001
skill_WORLD46218-90.1775050.6953660.6341240.1136120.2560710.000001

inference_result.xls

usermachine_nameutterance_start_timeutterance_smstarget_word_numberskillhelpknowledgecorrecttranscript_keytrn_correctasr_acceptconfidence_scoreasr_confidence
fCA8-5-1994-06-27LISTEN01-302-042004-11-09 11:25:242181HELLO10.801846NULLNULLNULL20.1247511
fCA7-5-1994-06-27LISTEN01-315-042005-01-24 11:33:124681HELLO10.997498NULLNULLNULL20.07851521

Additional Output

For efficiency/debug purposes, we have also output the following files for you convenience:

Project Listen

Training a Student Model to be used in the Reading Tutor

  1. Specify the Bayes nets in the property_xml file.
  2. Provide the data.
  3. Execute RunBnet.m
  4. Retrieve the model parameters from the param_table.
  5. Derive the equation for student models.
  6. Inference pknow and pcorrect using the equations derived by hand.
  7. Verify the correctness of your derivation by comparing the pknow and pcorrect you derived to what BNT-SM estimated in inference_result.

Training the Out of Vocabulary (OOV) model