Daniel B. Neill |
Dean's Career Development Professor
and Associate Professor of Information Systems
Director, Event and Pattern Detection Laboratory
H.J. Heinz III College
Carnegie Mellon University
Hamburg Hall #2105B, x8-3885
neill @ cs.cmu.edu
I am an Associate Professor of Information Systems in the Heinz College at Carnegie Mellon
University, where I hold the H.J. Heinz III College Dean's Career Development Professorship.
I also hold courtesy appointments in the Machine Learning Department and Robotics Institute in CMU's School of Computer Science, and an
adjunct appointment in the Department
of Biomedical Informatics at the University of Pittsburgh. I received my
Ph.D. in Computer Science from CMU in 2006. Before that, I received my
B.S.E. from Duke University, M.Phil. from Cambridge University, and M.S. from
Carnegie Mellon. At CMU, I direct the Event and Pattern Detection
Laboratory, and co-direct the Healthcare
Information Technology thrust of Heinz College's iLab.
LATEST NEWS: Our Event and Pattern Detection Laboratory has a new web site! Please update your pointers to
I am currently teaching four courses at the Heinz College. Course
descriptions, sample syllabi, and lecture slides can be obtained by
clicking on the course names below, and current course materials are
available on Blackboard.
Statistics for IT
Managers (95-796) is the core statistics course for students in the Master
of Information Systems Management program.
Large Scale Data Analysis for Public
Policy (90-866) is a master's level course which focuses on the
application of artificial intelligence and machine learning methods to
real-world policy problems.
I am also teaching two Ph.D.-level seminar courses, intended for doctoral
students (and qualified master's students) from Heinz College, the Machine
Learning Department, and other university departments who wish to engage
in cutting-edge research at the intersection of machine learning and
public policy. The Research Seminar in
Machine Learning and Policy (90-904, cross-listed in MLD as 10-830)
is a half-semester course which covers a broad range of MLP topics.
Special Topics in Machine Learning and Policy (90-921, cross-listed in MLD
as 10-831) is a half-semester course which will explore a single MLP topic
in detail. Topics covered include Event and Pattern Detection
(Spring 2010 and Spring 2014), Machine Learning for the Developing World (Spring
2011), Harnessing the Wisdom of Crowds (Spring 2012), and Mining Massive Datasets (Spring 2013).
I am also coordinating the new Joint Ph.D. Program in Machine Learning and
Public Policy, offered jointly by the Heinz College and Machine Learning
Department at CMU. Information about this program is available
Please see our new Event and Pattern Detection Laboratory web site, http://epdlab.heinz.cmu.edu, for the most up to date descriptions
of our ongoing research projects, and links to our publications and presentations.
My research is focused on novel statistical and computational methods for
discovery of emerging events and other relevant patterns in complex and
massive datasets, applied to real-world policy problems ranging from
medicine and public health to law enforcement and security. Application
areas include disease surveillance (e.g., using electronically
available public health data such as hospital visits and medication sales
to automatically identify and characterize emerging outbreaks), law
enforcement (e.g., detection and prediction of crime patterns using
offense reports and 911 calls), and health care (e.g., detecting
anomalous patterns of care which significantly impact patient
I was recently featured in IEEE
Intelligent Systems Magazine, as one of their "ten artificial intelligence
researchers to watch". A more detailed description of my research
(updated July 2011) is available here,
and my 2011 CSD/MLD immigration course talk is available here.
*** I am currently seeking Heinz and SCS Ph.D. students for research on
the following funded projects: ***
NSF IIS-0953330, CAREER: Machine Learning and Event Detection for the
Public Good (summary) (NSF
page) (project page).
NSF IIS-0916345, Fast Subset Scan for Anomalous Pattern Detection (summary) (NSF
page) (project page).
NSF IIS-0911032, Discovering Complex Anomalous Patterns (summary) (NSF
John D. and Catherine T. MacArthur Foundation, Evaluating Machine
Learning Methods and Tools for Use in Human Rights Work (with CMU's
Center for Human Rights Science). We will develop new machine learning
methods for early detection and advance prediction of conflict events,
and evaluate the potential utility of these methods for enabling
proactive responses to outbreaks of violence and human rights
Bloomberg Philanthropies, The Chicago SmartData Platform.
We are collaborating with the City of
Chicago to develop new methods for event prediction and apply these
methods to improve operations across all of the city's departments,
including law enforcement, public health, sanitation/rodent control,
emergency services, transportation, etc.
In general, my research focuses on the development of new statistical and
computational techniques for accurate and efficient pattern detection in
massive, high-dimensional datasets. While most previous data mining work
has focused on detection and classification of single records, pattern
detection extends these methods to groups of records, in order
to detect and identify patterns not visible from any individual record
alone. A key idea of our work is that pattern detection can often be
transformed into a subset scan problem, in which we search over
subsets of the data records to find those groups that are likely to
correspond to some probabilistically modeled pattern type. However, this
idea creates two main challenges: the statistical problem of evaluating
the "interestingness" of a given subset (whether it corresponds to some
specific pattern, is anomalous, etc.) and the computational problem of
efficiently searching a massive dataset for the most interesting subsets
(finding a "needle in the haystack").
Our past work has focused primarily on detection of
emerging events (e.g. outbreaks of disease) in multivariate spatial time
series data. We have developed a variety of new statistical methods which
achieve more timely and accurate event detection through better use of
spatial and temporal information, integration of multiple data streams,
and incorporation of prior knowledge.
Some current research topics include:
Primary application areas include disease surveillance, monitoring
of water quality and food safety, detection and prediction of crime
patterns, network intrusion detection, fraud detection, and scientific
discovery. We are currently involved in the development and deployment of
several large-scale systems for health and crime surveillance. These
collaborations will provide exciting opportunities to work with real-world
data, interact with law enforcement and public health officials, and
directly contribute to the public good by improving health, safety, and
- Extending event detection methodology to more general approaches for
pattern detection in large multivariate datasets.
- Developing novel Bayesian and nonparametric approaches for more
accurate detection, characterization, and explanation of events and
- Creating new, fast algorithms for computationally efficient detection
of patterns in massive datasets.
- Incorporating model learning into the event detection framework,
enabling us to distinguish between relevant and irrelevant patterns.
- Incorporating active learning from user feedback, enabling us to
rapidly "zero in" on those patterns that are most relevant to an
- Integrating web-scale data sources, such as search engine queries or
information from online social networks.
- Providing interactive tools for investigation, tracking, and
discovery of patterns in massive data.
Here are links to some recent papers. A complete list of publications is
available in my CV.
EVENT AND PATTERN DETECTION
Skyler Speakman, Edward McFowland III, and Daniel B. Neill. Scalable detection of anomalous
patterns with connectivity constraints. Journal of Computational and Graphical
Statistics, 2014, in press. (accepted author
Feng Chen and Daniel B. Neill. Non-parametric scan statistics for event detection and
forecasting in heterogeneous social media graphs. Proceedings of the 20th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining, 1166-1175, 2014. (pdf)
Edward McFowland III, Skyler Speakman, and Daniel B. Neill. Fast
generalized subset scan for anomalous pattern detection. Journal of Machine
Learning Research, 14: 1533-1561, 2013. (pdf)
Skyler Speakman, Yating Zhang, and Daniel B. Neill. Dynamic pattern detection with temporal consistency and
connectivity constraints. Proc. 13th IEEE International Conference on Data Mining, 697-706, 2013. (pdf)
Sriram Somanchi and Daniel B. Neill. Discovering anomalous patterns in large digital pathology images. Proc.
8th INFORMS Workshop on Data Mining and Health Informatics, 2013. (pdf)
Daniel B. Neill, Edward McFowland III, and Huanian Zheng. Fast subset
scan for multivariate event detection. Statistics in Medicine
32: 2185-2208, 2013. (pdf)
Daniel B. Neill. Fast subset scan for spatial pattern detection.
Journal of the Royal Statistical Society (Series B: Statistical
Methodology) 74(2): 337-360, 2012. (pdf)
Daniel B. Neill. New directions in artificial intelligence for public health
surveillance. IEEE Intelligent Systems 27(1): 56-59, 2012. (pdf)
Kan Shao, Yandong Liu, and Daniel B. Neill. A generalized fast subset
sums framework for Bayesian event detection. Proceedings of the 11th
IEEE International Conference on Data Mining, 617-625, 2011. (pdf)
Daniel B. Neill. Fast Bayesian scan statistics for multivariate event
detection and visualization. Statistics in Medicine 30(5):
455-469, 2011. (pdf)
Daniel Oliveira, Daniel B. Neill, James H. Garrett Jr., and Lucio
Soibelman. Detection of patterns in water distribution pipe breakage
using spatial scan statistics for point events in a physical network.
Journal of Computing in Civil Engineering 25(1): 21-30,
Daniel B. Neill and Gregory F. Cooper. A multivariate Bayesian scan
statistic for early event detection and characterization. Machine
Learning 79: 261-282, 2010. (pdf)
Daniel B. Neill and Weng-Keen Wong. A tutorial on event detection.
Presented at the 15th ACM SIGKDD Conference on Knowledge Discovery and
Data Mining, 2009. (pdf)
Daniel B. Neill. An empirical comparison of spatial scan statistics for
outbreak detection. International Journal of Health Geographics 8:
20, 2009. (pdf) (open
Daniel B. Neill. Expectation-based scan statistics for monitoring spatial
time series data. International Journal of Forecasting 25:
498-517, 2009. (pdf)
Daniel B. Neill, Gregory F. Cooper, Kaustav Das, Xia Jiang, and Jeff
Schneider. Bayesian network scan statistics for multivariate pattern
detection. In J. Glaz, V. Pozdnyakov, and S. Wallenstein, eds., Scan
Statistics: Methods and Applications, 221-250, 2009. (pdf)
Kaustav Das, Jeff Schneider, and Daniel B. Neill. Anomaly pattern
detection in categorical datasets. Proceedings of the 14th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining, 169-176, 2008.
Maxim Makatchev and Daniel B. Neill. Learning outbreak regions in
Bayesian spatial scan statistics. Proceedings of the ICML/UAI/COLT
Workshop on Machine Learning for Health Care Applications, 2008.
Daniel B. Neill. Detection of spatial and spatio-temporal clusters.
Ph.D. thesis, Carnegie Mellon University, Department of Computer
Science, Technical Report CMU-CS-06-142, 2006.
Daniel B. Neill, Andrew W. Moore, and Gregory F. Cooper. A
Bayesian spatial scan statistic. In Y. Weiss, et al., eds. Advances
in Neural Information Processing Systems 18, 1003-1010, 2006.
Daniel B. Neill, Andrew W. Moore, Maheshkumar Sabhnani, and Kenny
Daniel. Detection of emerging space-time clusters.
Proceedings of the 11th ACM SIGKDD Conference on Knowledge Discovery
and Data Mining, 218-227, 2005.
Daniel B. Neill and Andrew W. Moore. Anomalous spatial cluster
detection. Proceedings of the KDD 2005 Workshop on Data Mining
Methods for Anomaly Detection, 2005.
Daniel B. Neill, Andrew W. Moore, Francisco Pereira, and Tom Mitchell.
Detecting significant multidimensional spatial clusters. In L.K. Saul, et
al., eds. Advances in Neural Information Processing Systems 17,
Daniel B. Neill and Andrew W. Moore. Rapid detection of
significant spatial clusters. Proceedings of the 10th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining,
Xia Jiang, Gregory F. Cooper, and Daniel B. Neill. Generalized AMOC
curves for evaluation and improvement of event surveillance.
Proceedings of the American Medical Informatics Association Annual
Symposium, 281-285, 2009.
Maheshkumar R. Sabhnani, Daniel B. Neill, Andrew W. Moore, Fu-Chiang
Tsui, Michael M. Wagner, and Jeremy U. Espino. Detecting anomalous
patterns in pharmacy retail data. Proceedings of the KDD 2005
Workshop on Data Mining Methods for Anomaly Detection, 2005.
M. Wagner, F.-C. Tsui, J. Espino, W. Hogan, J. Hutman, J. Hersh, D. Neill,
A. Moore, G. Parks, C. Lewis, and R. Aller. A national retail data
monitor for public health surveillance. Morbidity and Mortality Weekly
Report 53: 40-42, 2004.
HEALTH CARE INFORMATION SYSTEMS
Daniel B. Neill. Using artificial intelligence to improve hospital inpatient care.
IEEE Intelligent Systems 28(2): 92-95, 2013. (pdf)
Christopher A. Harle, Daniel B. Neill, and Rema Padman. Information
visualization for chronic disease risk assessment. IEEE Intelligent
Systems 27(6): 81-85, 2012. (pdf)
Sharique Hasan, George T. Duncan, Daniel B. Neill, and Rema Padman.
Automatic detection of omissions in medication lists. Journal of the
American Medical Informatics Association 18(4): 449-458, 2011.
Huanian Zheng, Rema Padman, Sharique Hasan, and Daniel B. Neill. A
comparison of collaborative filtering methods for medication
reconciliation. Proceedings of the 13th International Congress on
Medical Informatics, 2010.
Sharique Hasan, George T. Duncan, Daniel B. Neill, and Rema Padman.
Towards a collaborative filtering approach to medication reconciliation.
Proceedings of the American Medical Informatics Association Annual
Symposium, 288-292, 2008.
Christopher A. Harle, Daniel B. Neill, and Rema Padman. An information
visualization approach to classification and assessment of diabetes risk
in primary care. Proceedings of the 3rd INFORMS Workshop on Data
Mining and Health Informatics, 2008.
Daniel B. Neill. Cascade effects in heterogeneous
populations. Rationality and Society 17(2): 191-241, 2005.
Daniel B. Neill. Evolutionary stability for large populations.
Journal of Theoretical Biology 227(3): 397-401, 2004.
Daniel B. Neill. Evolutionary dynamics with large aggregate
shocks. Dept. of Computer Science, Technical Report CMU-CS-03-197, 2003.
Daniel B. Neill. Cooperation and coordination in the Turn-Taking
Dilemma. Proceedings of the Ninth Conference on Theoretical Aspects
of Rationality and Knowledge: 231-244, 2003.
Daniel B. Neill. Optimality under noise: higher memory
strategies for the Alternating Prisoner's Dilemma. Journal of
Theoretical Biology 211(2): 159-180, 2001.
NATURAL LANGUAGE PROCESSING
Paul Hsiung, Andrew Moore, Daniel Neill, and Jeff Schneider.
Alias detection in link data sets. Proceedings of the First
International Conference on Intelligence Analysis, 2005.
Daniel B. Neill. Fully automatic word sense induction by
semantic clustering. Cambridge University, masters thesis, M.Phil. in
Computer Speech, 2002.
Event and Pattern Detection Laboratory
Arts and Letters Daily