next up previous contents
Next: Decisions with Locally Weighted Up: Blackbox Model Selection Previous: How long should Blackbox

Autonomous model selection pros and cons

Parameter tweaking is a tedious operation not always performed well by people, so automating it is an obvious plus. There are several other benefits from it too. Automation makes it possible for non-experts to apply machine learning. Automation also makes it possible to put machine learning in embedded systems where humans don't have the access required to do the tweaking.

There are several things to be careful of with autonomous model selection. It can be computationally expensive, which is an issue if a human expert would have been able to find the model more quickly than a machine doing a comparatively brute-force approach. It is necessary to guard against overfitting when doing lots of aggressive cross validation. Vizier protects itself from this by holding out an additional data set (CV Police). Cross validation minimizes the prediction error, but there are scenarios where minimizing the prediction error is not what is needed. For example, when data is sparse, it is desirable to choose broad kernel widths in order to get the most information from the available data, even though doing so may degrade cross validation prediction accuracy.

We believe the pros for autonomous model selection strongly outweigh the cons, and thus have made it central to Vizier's operation. However, Vizier also offers the flexibility and the tools to completely take over the model selection process for those experts that prefer to.



Jeff Schneider
Fri Feb 7 18:00:08 EST 1997