Selected Publications

BibTeX entries for all papers: settles.bib
More publications, citations, etc. on Google Scholar

• 
  Let's Get Together: The Formation and Success of Online Creative Collaborations
  B. Settles and S. Dow
  Human Factors in Computing Systems (CHI), to appear. ACM, 2013
  We study collaboration in an online music community, combining member surveys with a novel path-based regression analysis of the social network. We find that communication, compatible but complementary interests, and slight differences in status are key factors in collab formation; and that balanced efforts from both parties contribute to collab success. Our model also predicts new collaborations more accurately than standard baselines.
  pdf · video · blog post
• 
  Learning from Human-Generated Lists
  K.S. Jun, X. Zhu, B. Settles and T.T. Rogers
  International Conference on Machine Learning (ICML), to appear. 2013
  We propose sampling with reduced replacement (SWIRL), a new computational cognitive model of human list generation. SWIRL captures the importance of both order and repetition in such lists, which we further exploit as priors and constraints in machine learning algorithms (e.g., text classifiers trained from human-generated word lists), or use for classification directly (e.g., to predict cognitive dysfunction).
  
• 
  Active Learning   Book
  B. Settles
  Morgan & Claypool, 2012
  A short introductory text on active learning, a subfield of machine learning and artificial intelligence. For researchers, graduate students, and engineers working in computer science, statistics, psychology, and related fields.
  Publisher's website · Amazon.com · active-learning.net
• 
  Behavioral Factors in Interactive Training of Text Classifiers
  B. Settles and X. Zhu
  North American Association for Computational Linguistics - Human Language Technologies (NAACL HLT), pages 563-567. ACL, 2012
  As interactive annotation interfaces offer humans more expressive ways of "teaching" machine learning systems, what impact these varied annotation choices have? This paper looks at the effects of actions taken by human annotators on interactively-trained text classifiers.
  pdf
• 
  Closing the Loop: Fast, Interactive Semi-Supervised Annotation With Queries on Features and Instances
  B. Settles
  Empirical Methods in Natural Language Processing (EMNLP), pages 1467-1478. ACL, 2011
  DUALIST is a novel active learning paradigm which solicits and learns from labels on both features (e.g., words) and instances (e.g., documents). This setting motivates a new, fast, and flexible semi-supervised training algorithm for such dual supervision. Human annotators in user studies were able to produce near-state-of-the-art results with only a few minutes of effort.
  pdf · software
• 
  SIRT3 Substrate Specificity Determined by Peptide Arrays and Machine Learning   Cover Article
  B.C. Smith, B. Settles, W.C. Hallows, M.W. Craven, and J.M. Denu
  ACS Chemical Biology, 6(2):146-157, 2011
  SIRT3 is an important mitochondrial enzyme, linked to survivorship in diabetes and various age-related diseases. By using peptide screens of key potential binding sites as training data, we use machine learning to induce a model which (1) accurately predicts SIRT3 binding specificity, which we apply to the entire mitochondrial proteome to identify potential target substrates, and (2) is highly interpretable, leading to an improved understanding of the structure and function of SIRT3 and its chemical interactions.
  pdf · supporting info
• 
  Plugged in to the Community: Social Motivators in Online Goal-Setting Groups
  M. Burke and B. Settles
  International Conference on Communities & Technologies (C&T), pages 1-10. ACM, 2011
  We use computational models to examine two social factors in an online songwriting community challenge: (1) early feedback evoking a shared social identity, and (2) one-on-one collaborations with other members. We find that users who engage in these social features perform better at their goals than those who are non-social. We also begin to characterize the properties of "successful" collaborative interactions.
  pdf
• 
  Toward an Architecture for Never-Ending Language Learning
  A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E.R. Hruschka Jr. and T.M. Mitchell
  Conference on Artificial Intelligence (AAAI), pages 1306-1313. AAAI Press, 2010
  The architecture for NELL (never-ending language learner), a large-scale natural language processing system that runs continuously, 24x7, using multi-task semi-supervised learning methods to extract structured information from the World Wide Web.
  pdf · website · supporting info
• 
  Learning to Tag from Open Vocabulary Labels
  E. Law, B. Settles, and T.M. Mitchell
  European Conference on Machine Learning & Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), pages 211-226. Springer, 2010
  A new approach to classifying and retrieving media content, using "tags" from social websites and human computation systems as training data. Such labels are open-vocabulary and thus noisy and sparse, but we organize them into well-behaved semantic classes via topic modeling, and learn to predict these class distributions from media features. We demonstrate the scalability and accuracy of this approach on data collected from an online music annotation game, and also show the need for human evaluations in such open-vocabulary tasks.
  pdf
• 
  Computational Creativity Tools for Songwriters
  B. Settles
  NAACL-HLT Workshop on Computational Approaches to Linguistic Creativity, pages 49-57. ACL, 2010
  A new approach to classifying and retrieving media content, using "tags" from social websites and human computation systems as training data. Such labels are open-vocabulary and thus noisy and sparse, but we organize them into well-behaved semantic classes via topic modeling, and learn to predict these class distributions from media features. We demonstrate the scalability and accuracy of this approach on data collected from an online music annotation game, and also show the need for human evaluations in such open-vocabulary tasks.
  pdf · demos
• 
  Active Learning by Labeling Features
  G. Druck, B. Settles, and A. McCallum
  Empirical Methods in Natural Language Processing (EMNLP), pages 81-90. ACL, 2009
  In natural language tasks, features can often be intuitively labeled (e.g., in extracting information from apartment classifieds, "WORD=deposit" might indicate the label "lease," or "WORD=pets" indicate "restrictions"). We introduce novel query algorithms and user labeling interfaces for feature-based active learning in such domains.
  pdf
• 
  Active Learning Literature Survey
  B. Settles
  Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009
  A survey of active learning literature. See the book Active Learning above for an updated an more comprehensive treatment of this topic.
  pdf
• 
  Curious Machines: Active Learning with Structured Instances
  B. Settles
  PhD thesis, University of Wisconsin-Madison, 2008
  My PhD thesis on active learning for structured input representations (e.g., sequence labeling and multiple-instance learning tasks) and queries with potentially varying annotations costs. Also introduces the information density (ID) and expected gradient length (EGL) active learning frameworks.
  pdf
• 
  An Analysis of Active Learning Strategies for Sequence Labeling Tasks
  B. Settles and M. Craven
  Empirical Methods in Natural Language Processing (EMNLP), pages 1069-1078. ACL, 2008
  Active learning has not been well-studied for structured prediction tasks such as information extraction. This paper expands the frontier of query strategies for sequence models (CRFs, HMMs, PCFGs, etc.) into several new query frameworks, and presents a large-scale empirical evaluation of these algorithms on eight benchmark data sets.
  pdf · software
• 
  Active Learning with Real Annotation Costs
  B. Settles, M. Craven, and L. Friedland
  NIPS Workshop on Cost-Sensitive Learning, 2008
  Do annotation costs vary across instances? Among annotators? Can these costs be accurately predicted? What impact might this have on active learning in practice? This paper addresses these questions with a detailed empirical study of real-world annotations costs, and presents a novel approach to cost-sensitive active learning by modeling unknown annotation costs directly.
  pdf · data
• 
  Multiple-Instance Active Learning
  B. Settles, M. Craven, and S. Ray
  Advances in Neural Information Processing Systems (NIPS), volume 20, pages 1289-1296. MIT Press, 2008
  In multiple-instance (MI) learning, instances are organized into bags, which can be labeled inexpensively but ambiguously. In some MI problems, finer-granularity instance labels can be obtained, which are less ambiguous but more costly. This paper motivates a novel active learning framework for MI learners that allow them to query and learn from labels at mixed levels of granularity.
  pdf · code · data
• 
  Ranking Biomedical Passages for Relevance and Diversity
  A. Goldberg, D. Andrzejewski, J. Van Gael, B. Settles, X. Zhu and M. Craven
  Text Retrieval Conference (TREC), 2007
  An information retrieval system for biomedical text, focused on query generation and result ranking using a PageRank-style algorithm. The proposed ranker encourages both relevance and diversity in top ranked items, by turning retrieved items into absorbing states on a graph.
  pdf · code
• 
  Classifying Biomedical Articles by Making Localized Decisions
  T. Brow, B. Settles and M. Craven
  Text Retrieval Conference (TREC), 2006
  This paper presents a variety of machine learning approaches that exploit document-passage relationships both in classification and in learning. Results support our hypothesis that, for some text classification tasks, only certain passages of text are relevant to the task at hand.
  pdf
• 
  ABNER: An Open Source Tool for Automatically Tagging Genes, Proteins, and Other Entity Names in Text
  B. Settles
  Bioinformatics, 21(14):3191-3192. 2005
  An introduction to ABNER, a state-of-the-art, open-source, biomedical information extraction tool written in Java. It works stand-alone or as an API for inclusion in more sophisticated information management systems.
  pdf · software
• 
  Biomedical Named Entity Recognition Using Conditional Random Fields and Rich Feature Sets
  B. Settles
  International Joint Workshop on Natural Language Processing in Biomedicine and Its Applications (NLPBA), pages 104-107. 2004
  This paper motivates biomedical named entity recognition using conditional random fields (CRFs) with a variety of orthographic and automatically induced semantic features. It was one of the top performing approaches in the NLPBA shared task evaluation.
  pdf