NIPS*2000 Workshop:
Cross-Validation, Bootstrap and Model Selection

December 1, 2000: Beaver Run Resort (Peaks 9/10), Breckenridge, Colorado
Final schedule
NIPS*2000 Workshop Organizers:
Rahul Sukthankar, Compaq CRL and Robotics Institute, Carnegie Mellon
Larry Wasserman, Department of Statistics, Carnegie Mellon
Rich Caruana, Center for Automated Learning and Discovery, Carnegie Mellon


Cross-validation and bootstrap are popular methods for estimating generalization error based on resampling a limited pool of data, and have become widely-used for model selection. The aim of this workshop is to bring together researchers from both matchine learning and statistics in an informal setting to discuss current issues in resampling-based techniques. These include:

This one-day workshop will begin with an invited talk by Brad Efron, Professor of Statistics (Stanford) and inventor of the bootstrap algorithm. The remainder of the morning session will feature four talks focusing primarily on theoretical issues. The evening session will consist of four talks describing applications and new algorithms. Each 20 minute presentation will be followed by 10 minutes of discussion where interesting questions raised during the talk can be explored in some detail. Extended abstracts and other materials will be made available here, in advance so that workshop participants may prepare for the discussion sessions.

Call for Papers

The workshop submissions are closed. Accepted papers are listed in the schedule below.

Workshop Schedule

Morning Session
7:30-7:45Introduction (Rahul Sukthankar)
7:45-8:45Invited speaker: Brad Efron, The Cost of Model Selection
8:45-9:15Olivier Chapelle, Vladimir Vapnik, Yoshua Bengio: Model Selection for Small Sample Regression
9:15-9:45John Langford: PAC Bounds for Holdout Procedures
10:00-10:30Carl Rasmussen, Pedro Højen-Sørensen: Empirical Model Comparison: Bayesian Analysis of Disjoint Cross Validation for Continuous Losses
10:30-11:00Masashi Sugiyama, Hidemitsu Ogawa: Subspace Information Criterion - Unbiased Generalization Error Estimator for Linear Regression
Evening Session
16:30-17:00Bart Bakker, Tom Heskes: Model Clustering and Resampling
17:00-17:30Matthew Mullin, Rahul Sukthankar: Complete Cross Validation for Nearest Neighbor Classifiers
17:45-18:15Ashutosh Garg, Ira Cohen, Thomas Huang: Sampling Based EM Algorithm
18:15-18:45Koji Tsuda, Gunnar Rätsch, Sebastian Mika, Klaus-Robert Müller: Learning to Predict the Leave-one-out Error
18:45-19:00Wrap-up discussion

Additional Downloads

Extended abstracts are linked to the paper title above. This section contains pre-prints to full papers and additional related work.

Brad Efron talk slides
Olivier Chapelle, Vladimir Vapnik, Yoshua Bengio preprint
John Langford COLT-99 paper
Webpage with additional information (including NIPS presentation)
Masashi Sugiyama, Hidemitsu Ogawa Neural Computation article
Matthew Mullin, Rahul Sukthankar ICML paper

Contact Information

The workshop organizers can be contacted by email at, or at the phone/fax numbers listed below.
Rahul Sukthankar
Larry Wasserman
Rich Caruana

Rahul Sukthankar (
Last updated: November 26, 2000