Publications by Year

---

2012

• B. Settles. Active Learning. Morgan & Claypool, 2012.
  A short, self-contained introductory text on active learning, a subfield of machine learning and artificial intelligence. For researchers, graduate students, and engineers working in computer science and related fields.
• B. Settles and X. Zhu. Behavioral Factors in Interactive Training of Text Classifiers. Proceedings of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL HLT), short paper. ACL, 2012.
  As interactive annotation interfaces offer humans more expressive ways of "teaching" machine learning systems, what impact these varied annotation choices have? This paper looks at the effects of actions taken by human annotators on interactively-trained text classifiers. [paper]

2011

• B. Settles. Closing the Loop: Fast, Interactive Semi-Supervised Annotation With Queries on Features and Instances. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1467-1478. ACL, 2011.
  DUALIST is a novel active learning paradigm which solicits and learns from labels on both features (e.g., words) and instances (e.g., documents), motivating a new, fast, and flexible semi-supervised training algorithm for such dual supervision. Human annotators in user studies were able to produce near-state-of-the-art results with only a few minutes of effort. [paper][addendum][software]
• B.C. Smith, B. Settles, W.C. Hallows, M.W. Craven, and J.M. Denu. SIRT3 Substrate Specificity Determined by Peptide Arrays and Machine Learning. ACS Chemical Biology, 6(2):146-157. 2011.
  SIRT3 is an important mitochondrial enzyme, linked to survivorship in diabetes and various age-related diseases. By using peptide screens of key potential binding sites as training data, we use machine learning to induce a model which (1) accurately predicts SIRT3 binding specificity, which we apply to the entire mitochondrial proteome to identify potential target substrates, and (2) is highly interpretable, leading to an improved understanding of the structure and function of SIRT3 and its chemical interactions. [paper][supporting info]
  ⇒ Featured Article: On the Cover, ACS Podcast
• M. Burke and B. Settles. Plugged in to the Community: Social Motivators in Online Goal-Setting Groups. Proceedings of the International Conference on Communities & Technologies (C&T), pages 1-10. ACM, 2011.
  We induce computational models that examine two social factors in an online "songwriting challenge" community: (1) early feedback evoking a shared social identity, and (2) one-on-one collaborations with other members. We find that users who engage in these social features perform better at their goals than those who are non-social. We also begin to characterize the properties of "successful" collaborative interactions. [paper]
• E. Law, B. Settles, A. Snook, H. Surana, L. von Ahn and T. Mitchell. Human Computation for Attribute and Attribute Value Acquisition. Proceedings of the CVPR Workshop on Fine-Grained Visual Categorization. 2011.
  A short paper describing Polarity, a human computation game designed to actively elicit features and feature values from the crowd, which can be used to train machine learning algorithms. [paper]
• B. Settles. Algorithms for Active Learning. In B. Krishnapuram, S. Yu, and R.B. Rao (Eds.), Cost-Sensitive Machine Learning. Chapman and Hall, 2011.
  A gentle overview/tutorial on active learning algorithms and applications. Part of a larger textbook on cost-sensitive machine learning methods including semi-supervised, multi-task, multiple-instance, and online learning approaches (among others). [Amazon] [publisher's link]
• B. Settles. From Theories to Queries: Active Learning in Practice. JMLR Workshop and Conference Proceedings, 16:1-18. 2011.
  This review article identifies and discusses six current directions in active learning research aimed at making it more practical for real-world use. Invited paper for the Proceedings of the AISTATS 2010 Workshop on Active Learning and Experimental Design. [paper]

2010

• A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E.R. Hruschka Jr. and T.M. Mitchell. Toward an Architecture for Never-Ending Language Learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pages 1306-1313. AAAI Press, 2010.
  The architecture for NELL (never-ending language learner), a large-scale natural language processing system that runs continuously, 24x7, using multi-task semi-supervised learning methods to extract structured information from the world wide web. [paper][supplementary data]
  ⇒ Press Coverage: The New York Times, Universe, Word of Mouth
• B. Settles. Computational Creativity Tools for Songwriters. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Linguistic Creativity, pages 49-57. ACL, 2010.
  This paper presents two "computational creativity tools" that employ machine learning and natural language processing to assist songwriters, along with two general principles for the design of such tools. Empirical and anecdotal results from an international songwriting contest are presented. [paper][demos]
  ⇒ Press Coverage: The Associated Press, Marketplace, American Songwriter
• E. Law, B. Settles, and T.M. Mitchell. Learning to Tag from Open Vocabulary Labels. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), pages 211-226. Springer, 2010.
  A new approach to classifying and retrieving media content, using "tags" from social websites and human computation systems as training data. Such labels are open-vocabulary and thus noisy and sparse, but we organize them into well-behaved semantic classes via topic modeling, and learn to predict these class distributions from media features. We demonstrate the scalability and accuracy of this approach on data collected from an online music annotation game, and also show the need for human evaluations in such open-vocabulary tasks. [paper]

2009

• G. Druck, B. Settles, and A. McCallum. Active Learning by Labeling Features. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 81-90. ACL, 2009.
  In natural language tasks, features can often be intuitively labeled (e.g., in extracting information from apartment classifieds, "WORD=deposit" might indicate the label "lease," or "WORD=pets" indicate "restrictions"). We introduce novel query algorithms and user labeling interfaces for feature-based active learning in such domains. [paper]
• B. Settles. Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison. 2009.
  An introduction to active learning and a survey of the literature. This paper outlines the various learning scenarios, query strategy frameworks, variants, application domains, and related work published over the past few decades. [paper]
• B. Settles. A Software Tool for Biomedical Information Extraction (and Beyond). In V. Prince and M. Roche (Eds.), Information Retrieval in Biomedicine: Natural Language Processing for Knowledge Integration, pages 326-335. IGI Global Press, 2009.
  An overview of biomedical named entity recognition with conditional random fields using ABNER (see the Bioinformatics 2005 paper below). Includes a survey of higher-level information management tasks (relation extraction, information retrieval, automatic database curation, etc.) that have been built on top of ABNER. [Amazon]

2008

• B. Settles. Curious Machines: Active Learning with Structured Instances. PhD thesis, University of Wisconsin-Madison. 2008.
  My PhD thesis on active learning for structured input representations (e.g., sequence labeling and multiple-instance learning tasks) and queries with potentially varying annotations costs. Also introduces the information density (ID) and expected gradient length (EGL) active learning frameworks. [paper]
• B. Settles and M. Craven. An Analysis of Active Learning Strategies for Sequence Labeling Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1069-1078. ACL, 2008.
  Active learning has not been well-studied for structured prediction tasks such as information extraction. This paper expands the frontier of query strategies for sequence models (CRFs, HMMs, PCFGs, etc.) into several new query frameworks, and presents a large-scale empirical evaluation of these algorithms on eight benchmark data sets. [paper][code]
  
• B. Settles, M. Craven, and L. Friedland. Active Learning with Real Annotation Costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning. 2008.
  Do annotation costs vary across instances? Among annotators? Can these costs be accurately predicted? What impact might this have on active learning in practice? This paper addresses these questions with a detailed empirical study of real-world annotations costs, and presents a novel approach to cost-sensitive active learning by modeling unknown annotation costs directly. [paper][data]
• B. Settles, M. Craven, and S. Ray. Multiple-Instance Active Learning. In Advances in Neural Information Processing Systems (NIPS), volume 20, pages 1289-1296. MIT Press, 2008.
  In multiple-instance (MI) learning, instances are organized into bags, which can be labeled inexpensively but ambiguously. In some MI problems, finer-granularity instance labels can be obtained, which are less ambiguous but more costly. This paper motivates a novel active learning framework for MI learners that allow them to query and learn from labels at mixed levels of granularity. [paper][code][data]
  

2007

• A. Goldberg, D. Andrzejewski, J. Van Gael, B. Settles, X. Zhu and M. Craven. Ranking Biomedical Passages for Relevance and Diversity. In Proceedings of the Fifteenth Text Retrieval Conference (TREC). 2007.
  An information retrieval system for biomedical text, focused on query generation and result ranking using a PageRank-style algorithm. The proposed ranker encourages both relevance and diversity in top ranked items, by turning retrieved items into absorbing states on a graph. [paper][code].

2006

• T. Brow, B. Settles and M. Craven. Classifying Biomedical Articles by Making Localized Decisions. In Proceedings of the Fourteenth Text Retrieval Conference (TREC). 2006.
  This paper presents a variety of machine learning approaches that exploit document-passage relationships both in classification and in learning. Results support our hypothesis that, for some text classification tasks, only certain passages of text are relevant to the task at hand. [paper].

2005

• B. Settles. ABNER: An Open Source Tool for Automatically Tagging Genes, Proteins, and Other Entity Names in Text. Bioinformatics, 21(14):3191-3192. 2005.
  An introduction to ABNER, a state-of-the-art, open-source, biomedical information extraction tool written in Java. It works stand-alone or as an API for inclusion in more sophisticated information management systems. [paper][software]
• B. Settles and M. Craven. Exploiting Zone Information, Syntactic Features, and Informative Terms in Gene Ontology Annotation from Biomedical Documents. In Proceedings of the Thirteenth Text Retrieval Conference (TREC). 2005.
  A system that predicts Gene Ontology (GO) annotations for research articles using a two-tier machine learning approach. First, articles are segmented into "zones" (abstract, introduction, conclusion, etc.) and classified using automatically induced syntactic and semantic features. Second, zone-level predictions are aggregated into overall document labelings. This was one of the top performing systems at the TREC Genomics track. [paper]

2004

• B. Settles. Biomedical Named Entity Recognition Using Conditional Random Fields and Rich Feature Sets. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications (NLPBA), pages 104-107. 2004.
  This paper motivates biomedical named entity recognition using conditional random fields (CRFs) with a variety of orthographic and automatically induced semantic features. It was one of the top performing approaches in the NLPBA shared task evaluation. [paper]