In this section we review the use of techniques from Optimal Experiment Design (OED) to minimize the estimated variance of a neural network [Fedorov 1972,MacKay 1992,Cohn 1994]. We will assume we have been given a learner , a training set and a parameter vector estimate that maximizes some likelihood measure given . If, for example, one assumes that the data were produced by a process whose structure matches that of the network, and that noise in the process outputs is normal and independently identically distributed, then the negative log likelihood of given is proportional to
The maximum likelihood estimate for is that which minimizes .
The estimated output variance of the network is
where the true variance is approximated by a second-order Taylor series expansion around . This estimate makes the assumption that is locally linear. Combined with the assumption that is Gaussian with constant variance for all x, one can derive a closed form expression for . See [Cohn ] for details.
In practice, may be highly nonlinear, and may be far from Gaussian; in spite of this, empirical results show that it works well on some problems [Cohn 1994]. It has the advantage of being grounded in statistics, and is optimal given the assumptions. Furthermore, the expectation is differentiable with respect to . As such, it is applicable in continuous domains with continuous action spaces, and allows hillclimbing to find the that minimizes the expected model variance.
For neural networks, however, this approach has many disadvantages. In addition to relying on simplifications and assumptions which hold only approximately, the process is computationally expensive. Computing the variance estimate requires inversion of a matrix for each new example, and incorporating new examples into the network requires expensive retraining. Paass and Kindermann paass-kindermann95 discuss a Markov-chain based sampling approach which addresses some of these problems. In the rest of this paper, we consider two ``non-neural'' machine learning architectures that are much more amenable to optimal data selection.