In many domains such as speech recognition and machine translation it is extremely useful to be able able to distinguish coherent from non-coherent sentences. We introduce a set of word-based statistical features which measure semantic coherence and can be used to enhance any language application where coherent sentences need to be generated or recognized. We train a decision tree using the constructed feature set to automatically classify sentences as coherent or not. We find that our combination of boosted decision trees and coherence features achieves an accuracy of 80\% when distinguishing trigram-generated sentences (non-coherent) from those in the Broadcast News dataset (coherent).