Decision Trees

Key concepts:

1. Building decision trees
2. Evaluating decision trees
3. Pruning decision trees

Building Decision Trees

Decision trees are tree-structured models for classification and regression.

The figure below shows an example of a decision tree to determine what kind of contact lens a person may wear. The choices (classes) are none, soft and hard. The attributes that we can obtain from the person are their tear production rate (reduced or normal), whether they have astigmatism (yes/no), their age category (presbyopic or not, young or not), their spectacle prescription (myopia or hypermetropia).

Decision trees can be learned from training data. Training data will typically comprise many instances of the following kind:

Instance 1	attribute1	attribute2	...	attributeK	class
Instance 2	attribute1	attribute2	...	attributeK	class

The decision tree learning algorithm recursively learns the tree as follows:

1. Assign all training instances to the root of the tree. Set curent node to root node.
2. For each attribute
   1. Partition all data instances at the node by the value of the attribute.
   2. Compute the information gain ratio from the partitioning.
3. Identify feature that results in the greatest information gain ratio. Set this feature to be the splitting criterion at the current node.
   • If the best information gain ratio is 0, tag the current node as a leaf and return.
4. Partition all instances according to attribute value of the best feature.
5. Denote each partition as a child node of the current node.
6. For each child node:
   1. If the child node is “pure” (has instances from only one class) tag it as a leaf and return.
   2. If not set the child node as the current node and recurse to step 2.

The following pseudo code describes the procedure. The pseudocode is a bit more detailed than your usual pseudo code, and doesn't follow any known standard :-)

In the pseudocode class variables are prefixed by “@” to distinguish them from locla varibles.

Pseudocode to train a decision tree

#------- Some "Helper" functions ---------------
# Segregating out instances that take a particular value
# attributearray is an N x 1 array.
def segregate(attributearray, value):
    outlist = []
    for i = 1 to length(attributearray):
        if (attributearray[i] == value):
            outlist = [outlist, i]  #  Append "i" to outlist
    return outlist


# Assuming labels take values 1..M.
def computeEntropy(labels):
    entropy = 0
    for i = 1 to M:
        probability_i = length(segregate(labels, i)) / length(labels)
        entropy -= probability_i * log(probability_i)
    return entropy
  
# Find most frequent value. Assuming labels take values 1..M 
def mostFrequentlyOccurringValue(labels):
    bestCount = -inf
    bestId = none
    for i = 1 to M:
        count_i = length(segregate(label,i))
        if (count_i > bestCount):
            bestCount = count_i
            bestId = i
    return bestId

#-------- The Dtree code ------------

#Here "attributes" is an Num-instance x Num-attributes matrix. Each row is
#one training instance.
#"labels" is a Num-instance x 1 array of class labels for the training instances

# Note, we're storing a number of seemingly unnecessary variables, but
# we'll use them later for counting and pruning
class  dtree:
    float @nodeGainRatio
    float @nodeInformationGain
    boolean @isLeaf 
    integer @majorityClass
    integer @bestAttribute
    dtree[] @children
    dtree   @parent

    init(attributes, labels):
        @parent = null
        buildTree (attributes, labels, self)

    buildTree (attributes, labels, self):
        numInstances = length(labels)
        nodeInformation = numInstances * computeEntropy(labels)
        @majorityClass = mostFrequentlyOccurringValue(labels)

        if (nodeinformation == 0):  # This is a "pure" node
            @isLeaf = True
            return

        # First find the best attribute for this node
        bestAttribute = none
        bestInformationGain = -inf
        bestGainRatio = -inf
        for each attribute X:
            conditionalInfo = 0
            attributeEntropy = 0
            for each attributevalue Y:
                ids = segregate(attributes[][X], Y) # get ids of all instances
                                                    # for which attribute X == Y
 
                attributeCount[Y] = length(ids)
                conditionalInfo += attributeCount[Y] * computeEntropy(labels(ids));
            attributeInformationGain =  nodeInformation - conditionalInfo
            gainRatio = attributeInformationGain / computeEntropy(attributeCount)
            if (gainRatio > bestGainRatio):
                bestInformationGain = attributeInformationGain
                bestGainRatio = gainRatio
                bestAttribute = X

        #If no attribute provides andy gain, this node cannot be split further
        if (bestGainRatio == 0):
            @isLeaf = True
            return

        # Otherwise split by the best attribute
        @bestAttribute = bestAttribute
        @nodeGainRatio = bestGainRatio
        @nodeInformationGain = bestInformationGain
        for each attributevalue Y:
            ids = segregate(attributes[][bestAttribute], Y)
            @children[Y] = dtree(attributes[ids], labels[ids])
            @children[Y].@parent = self
        return
        

Evaluating an instance using a decision tree

Once a decision tree is learned, it can be used to evaluate new instances to determine their class. The instance is passed down the tree, from the root, until it arrives at a leaf. The class assigned to the instance is the class for the leaf.

This procedure is explained by the following pseudocode.

Pseudocode to evaluate a decision tree

class dtree:
    [All the stuff from before here]

    def evaluate(testAttributes):
        if (@isLeaf):
            return @majorityClass
        else
            return @children[testAttributes[@bestAttribute]].evaluate(testAttributes)

Pruning decision trees

Decision trees that are trained on any training data run the risk of overfitting the training data.

What we mean by this is that eventually each leaf will reperesent a very specific set of attribute combinations that are seen in the training data, and the tree will consequently not be able to classify attribute value combinations that are not seen in the training data.

In order to prevent this from happening, we must prune the decision tree.

By pruning we mean that the lower ends (the leaves) of the tree are “snipped” until the tree is much smaller. The figure below shows an example of a full tree, and the same tree after it has been pruned to have only 4 leaves.

Pruning can be performed in many ways. Here are two.

Pruning by Information Gain

The process of IG-based pruning requires us to identify “twigs”, nodes whose children are all leaves. “Pruning” a twig removes all of the leaves which are the children of the twig, and makes the twig a leaf. The figure below illustrates this.

The algorithm for pruning is as follows:

1. Catalog all twigs in the tree
2. Count the total number of leaves in the tree.
3. While the number of leaves in the tree exceeds the desired number:
   1. Find the twig with the least Information Gain
   2. Remove all child nodes of the twig.
   3. Relabel twig as a leaf.
   4. Update the leaf count.

The pseudocode for this pruning algorithm is below.

# Count leaves in a tree
def countLeaves(decisiontree):
    if decisiontree.isLeaf:
        return 1
    else
        n = 0
        for each child in decisiontree.children:
            n += countLeaves(child)
    return n


# Check if a node is a twig
def isTwig(decisionTree)
    for each child in decisiontree.children:
        if not child.isLeaf:
            return False
    return True


# Make a heap of twigs.  The default heap is empty
def collectTwigs(decisionTree, heap=[])
    if isTwig(decisionTree):
        heappush(heap,(decisionTree.@nodeInformationGain, decisionTree))
    else
        for each child in decisiontree.children:
            collectTwigs(child,heap)
    return heap


# Prune a tree to have nLeaves leaves
# Assuming heappop pops smallest value
def prune(dTree, nLeaves):
    totalLeaves = countLeaves(dTtree)
    twigHeap = collectTwigs(dTree)
    while totalLeaves > nLeaves:
        twig = heappop(twigHeap)
        totalLeaves -= (length(twig.@children) - 1) #Trimming the twig removes
                                                    #numChildren leaves, but adds
                                                    #the twig itself as a leaf
        twig.@chidren = null  # Kill the chilren
        twig.@isLeaf = True
        twig.@nodeInformationGain = 0

        # Check if the parent is a twig and, if so, put it in the heap
        parent = twig.@parent
        if isTwig(parent):
            heappush(twigHeap,(parent.@nodeInformationGain, parent))
    return

Pruning by Classification Performance on Validation Set

An alternate approach is to prune the tree to maximize classification performance on a validation set (a data set with known labels, which was not used to train the tree).

We pass the validation data down the tree. At each node, we record the total number of instances and the number of misclassifications, if that node were actually a leaf. We do this at all nodes and leaves.

Subsequently, we prune all twigs where pruning results in the smallest overall increase in classification error.

The overall algorithm for pruning is as follows:

Stage 1:

1. For each instance of validation data:
   Recursively pass

# First pass : evaluate validation data and note the classification at each node
# Assuming that "valData" includes "attributes" and "labels"

# First create an empty list of error counts at nodes
def createNodeList(dTree, nodeError=[]):
    nodeError[dTree] = 0
    for child in dTree.@children
        createNodeList(dTree, nodeError)
    return nodeError

# Pass a single instance down the tree and note node errors
def classifyValidationDataInstance(dTree, validationDataInstance, nodeError):
    if (dTree.@majorityClass != validationDataInstance.label):
        nodeError[dTree] += 1
    if (not @isLeaf):
        childNode = dTree.@children[testAttributes[@bestAttribute]]
        classifyValidationDataInstance(childNode, testAttributes, nodeError)
    return 

# Count total node errors for validation data
def classifyValidationData(dTree, validationData):
    nodeErrorCounts = createNodeList(dTree)
    for instance in validationData:
        classifyValidationDataInstance(child, instance, nodeErrorCounts)
    return nodeErrorCounts


# Second pass:  Create a heap with twigs using nodeErrorCounts
def collectTwigsByErrorCount(decisionTree, nodeErrorCounts, heap=[])
    if isTwig(decisionTree):
        # Count how much the error would increase if the twig were trimmed
        twigErrorIncrease = nodeErrorCounts[decisionTree]
        for child in decisionTree.@children
            twigErrorIncrease -= nodeErrorCounts[child]
        heappush(heap,(twigErrorIncrease, decisionTree))
    else
        for each child in decisiontree.children:
            collectTwigsByErrorCount(child, nodeErrorCounts, heap)
    return heap



# Third pass: Prune a tree to have nLeaves leaves
# Assuming heappop pops smallest value
def pruneByClassificationError(dTree, validationData, nLeaves):
    # First obtain error counts for validation data
    nodeErrorCounts = classifyValidationData(dTree, validationData)
    # Get Twig Heap
    twigHeap = collectTwigsByErrorCount(dTree, nodeErrorCounts)

    totalLeaves = countLeaves(dTtree)
    while totalLeaves > nLeaves:
        twig = heappop(twigHeap)
        totalLeaves -= (length(twig.@children) - 1) #Trimming the twig removes
                                                    #numChildren leaves, but adds
                                                    #the twig itself as a leaf
        twig.@chidren = null  # Kill the chilren
        twig.@isLeaf = True
        twig.@nodeInformationGain = 0

        # Check if the parent is a twig and, if so, put it in the heap
        parent = twig.@parent
        if isTwig(parent):
            twigErrorIncrease = nodeErrorCounts[parent]
            for child in parent.@children
                twigErrorIncrease -= nodeErrorCounts[child]
            heappush(twigHeap,(twigErrorIncrease, parent))
    return