Predicting Social Modes of Knowledge Co-construction for Collaborative Learning	
	
Yi-Chia Wang
Language Technologies Institute
School of Computer Science
Carnegie Mellon University
yichiaw@cs.cmu.edu
	
Abstract
Computer-supported collaborative learning (CSCL) mainly aims at engaging students in discourse activities which are related to knowledge acquisition. Recently, some CSCL researchers focus on analyzing process dimensions of knowledge construction in order to assist teachers in adjusting instruction guidelines. In this paper, we propose approaches for detecting social modes of argumentative knowledge construction (SOC) by drawing the information concealed between parent and current message in the same thread into consideration. Moreover, we introduce the extracted features into sequential learning algorithms since it is reasonable for us to hypothesize that there are some correlation between posted articles which appear the same discourse threads. Finally, some experiments were conducted to show that our presented feature selection approaches would benefit existing classification learning algorithms . 
1	Introduction and Related Work
Computer-supported collaborative learning (CSCL) often aims at engaging students in argumentative discourse activities with the goal of knowledge acquisition. In CSCL environment, students often communicate their ideas of an assigned problem through asynchronous and mostly text-oriented discussion boards. For example, an individual student may post his opinions or analyses of a problem on a discussion board, and other learning partners may read the message and reply it with agreements, questions, and so on. 
Although students in one learning group collaboratively construct a text during CSCL, not each of them contributes in the same degree to knowledge acquisition. However, it would be valuable for teachers to adjust instruction direction if we can identify each student’s contribution degree in one group.
Motivated by this, we want to predict the social modes of co-construction (SOC) for posted messages.  Here, SOC indicate different degrees to which students refer to contributions of their learning partners [1]. There are five major categories of SOC, including externalization, elicitation, quick consensus building, integration-oriented consensus building, and conflict-oriented consensus building.
In our previous work, Manjinder (2006) constructed a finite state automaton to predict SOC codes. The automaton is designed to make transitions between states whenever an input message segment has some specified features, such as quote marks and prompts. However, there is no salient improvement on SOC prediction.
By examining definitions of SOC categories, we find that it is quite reasonable to hypothesize that the relation between parent and current message in a thread structure would affect the SOC code of current message. In the current work, we would like to consider thread structure as one of features in the feature space. Also, by applying sequential learning [2, 3], we are able to take preceding information into consideration. This should also benefit the SOC prediction. 
2	SOC Prediction Approaches
We first define the problem which we are going to solve more specifically. Figure 1 is an example of a thread which could appear on a discussion board. The top black node represents the initial message of a thread, which is followed by replied messages. In our task, we assume that each message is divided into several segments. For each segment, it is assigned a SOC code. Our goal is to detect SOC for every message segment given the whole thread structure on a discussion board.

Segment 1:  <SOC>
Segment 2:  <SOC>
:
Segment n: <SOC>

Figure 1: An illustration of a discussion thread.
2.1	Feature Space Representation
Our introduced feature space for each message contains two major parts. One is syntactic information encoded in messages. Another part would consider the specific features which appear in hierarchical discussion board. We explain the details as following:
To represent the information of message segments, we identify each segment as a vector instance in our vector-space. Every segment was represented as a vector x = <a1, …, an, p1, …pn, s, pmt, quo, len, fsm, deep>, which is concatenated by three different parts of information extracted from the corresponding segment. 
The first part, a = <a1, …, an>, denotes a vector of terms (unigrams, bigrams, or part-of-speech tags) of current segment. The reason why we bring the syntactic information of messages into the feature space is that we believe sometimes some specific words could have strong associations for us to identify a particular SOC category. Taking the Elicitation category in SOC as an example, since it describes the social modes of questioning the learning partner or provoking a reaction from the learning partner, we can intuitively have ideas that question words, such as who, why, should appear more frequently in this category than others.
Second, we try to utilize some characteristics particularly related to thread hierarchical structure of discussion board. Here, several different meta features are considered. The first one attempts to capture the similarity between parent and child node. To achieve this goal, we consider the term overlap between current segment and its parent node in the same thread by introducing two feature vectors p and s into x. The vector p = <p1, …, pn> represents the overlapping between current segment vector ac and parent message vector ap. Finally, s computes the similarity (ex: cosine similarity) between a and p. A resulting set of message segments would be S = {(x1, y1), (x2, y2), …, (xm, ym)}, where label yi is the SOC code for segment xi.
In addition to add the cosine similarity between parent and child message into our feature space, there are five more meta features, pmt, quo, len, fsm, deep, which we also concern about. Our belief is that this kind of information should be representative of the role which a message plays in constructing argumentative knowledge. Feature pmt identifies whether the current message is a prompting one which would be marked with ‘-’ punctuation in the beginning of message. Feature quo denotes a cited message original from previous one. We include these two features primarily because they can stand for different functions of messages during discourse conversation. 
The information of length of a message is coded in feature len which can be the opinion that assess the information accompanying with this particular message in some degree. 
Moreover, the feature fsm represents the states of a simple automaton which only has two states. The automaton is set to initial state (q0) whenever a new message segment starts. It makes a transition to state (q1) when it encounters a quoted message. In the state (q1) the automaton remains in this state for all the other cases except when it encounters a prompt. On encountering a prompt it makes a transition back to the initial state (q0). This feature is used to indicate whether the current message belongs to some particular layers in this “fold” discourse structure.
Finally, we include the deep feature to identify how depth a message is now located in its belonging thread. To summarize, except for the feature len, it is obvious that all the other four introduced features carry different information related to tree structure. In Chapter 3, we will present and discuss the experimental results before and after including these features into a basic syntactic feature space.
2.2	Learning Algorithms for SOC Detection
In this paper, in order to demonstrate that our proposed feature set is quite representative of capturing characteristics for predicting SOC codes and can be applied to improve the performance of various classifying algorithms, we choose three different learning algorithms, Sequential Minimal Optimization (SMO), Naïve Bayes, and Decision Tree (J48 or C4.5), which are representative of various machine learning approaches and commonly used by natural language processing research.
SMO is a fast version of support vector machine learning algorithm. Simply saying, it reduces the training time by divide a large and time-consuming problem into small sequential pieces and solve each small problems one by one [4]. 
Naïve Bayes classifier is a probabilistic model which takes Bayes’ theorem as its theoretical basis. Although it assumes that every feature is independent and mostly we know this assumption is not the case in practice, it still works well in solving many research topics.
The third learning algorithm we are going to use is Decision Tree, which is a method for approximating discrete value classifier. Its pruning methodology makes it robust to noisy data.
2.3	Integrating Proposed Features with Sequential Learning Algorithm
The motivation which makes us to consider applying our presented feature set to sequential learning algorithm comes out from the idea that the sequential property among thread structure of discussion forums. More specifically, it seems reasonable that a following message segment would be related to its previous segment. Generally, given the SOC of a segment is known, one would expect that there must be some correlation between SOC codes of it and its following segment which appears in the same message. 
Based on this assumption, we arrange an experiment and try to see whether we can gain benefits from both meta features of thread structure and the sequential property of each individual message by applying Colli algorithm, which is one of sequential learning algorithm. The analyses result will be shown in next chapter.
3	Experiments
In this chapter, we will describe machine learning packages used, our measurement metric for evaluation, the data source, and the experimental results for different setups, respectively. 
For the preprocessing step, in addition to some programs written by myself to generate the appropriate input formats for learning tools, I also use TagHelper tool to automatically analyze the data set and do feature selection. After that, I utilized the two machine learning packages, Weka and MinorThird, as my learning tools to train the SOC predicting module and obtain some experimental results. 
3.1	Data Source
I will utilize a German discourse as my data set. The corpus was collected in an experimental online learning environment where students collaborated via web discussion boards. The data set has been processed into xml format and also tagged, including 21 different SOC labels for my usage, and evaluated by human. Figure 2 shows the distribution of SOC labels in the data set.
3.2	Evaluation Metric
Cohen’s Kappa Statistic is a commonly used metric in assessing inter-rater reliability on the defined analyses units among target categories. In our case of detecting SOC codes, the golden standard of Kappa is 0.81. The SOC discourse segments are rated according to how learners study together on the assigned task. 

Figure 2: Distribution of SOC in our data set
3.3	Initial Results
Figure 3 is the baseline results considering four different syntactic characteristics as basic feature set without taking any structure information available from discussion boards into concern yet. Although I used a 10-fold cross validation as a test set, we can still see the Kappa statistics are low. Therefore, one can see that it is insufficient to only consider the syntactic information of current message in order to detect its SOC codes. Moreover, we find that the unigram case works better than others. One reasonable explanation for this phenomenon would be probably because of the small amount data set we use which is not enough to capture higher level syntactic information such as bigram part-of-speech.
    
Figure 3: Performance of four different feature bases for top-100 rank features (SMO)

 Figure 4: Performance of three learning algorithms (unigram)
The above figure (4) describes the compared results for the three algorithms we have mentioned in previous chapter. By selecting top 100 unigrams as features, we can conclude that SMO has the highest Kappa value (0.47) among there three learning algorithms. 
3.4	Further Results
In order to demonstrate our proposed meta features (thread structure) can increase Kappa statistics in predicting SOC, we compared two classifiers. One is only trained with unigrams and another is trained with unigrams and also the meta features. For both cases, we performed feature selection before training phase and only chose top 100 features. Results are shown in Figure 5. We can see Kappa value increases from 0.47 to 0.59. This tells us that adding the meta features does help to improve the performance on predicting SOC codes.
The second experiment implemented a sequential learning algorithm, Colli, which is provided in the machine learning package, Minorthird. Although we can observe the improvement benefits from thread structure, the highest Kappa value (0.45) is still lower than the one in Figure 5 (0.59). Analyzing the causes, we think there may be two reasons. First, inefficient feature selection method could be one. Moreover, for a message segment, the most relative segment which would affect its SOC may be its parent’s instead of its previous segment in the same message.

Figure 5: Performance comparison after considering the information extracted from thread structure (SMO)

Figure 6: Performance comparison by considering including the information extracted from thread structure into sequential learning algorithm
4	Conclusion and Future Work
In this paper, we proposed approaches for detecting social modes of argumentative knowledge construction (SOC) by drawing the correlation concealed between parent and current message in the same thread into consideration. We showed that our selected meta feature indeed can improve the performance of various learning algorithms, including both non-sequential and sequential ones. However, the low Kappa statistics of applying the sequential algorithm to detect SOC is unexpected since we thought that the sequential property should be quite meaningful to help improve the predicting performance. To find out the reasons, some experiments have to be conducted.
References
[1] Weinberger, A. & Fischer, F. (2006) A Framework to Analyze Argumentative Knowledge Construction in Computer-Supported Collaborative Learning. Computers & Education 46(1):71-95.
[2] Cohen, W.W. & Carvalho, V.R. (2005). Stacked sequential learning. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-2005), pp. 671–676. Edinburgh, Scotland, UK.
[3] Lafferty, J., McCallum, A. & Pereira, F. (2001) Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the International Conference on Machine Learning (ICML-2001). Williams, MA.
[4] J. Platt. (1998). Sequential Minimal Optimization: A Fast Algorithm for Training Support Vector Machine. Microsoft Research Technical Report MSR-TR-98-14, (1998).