Neural networks are functions used to convert raw data into useful information. They are represented by models with many parameters. Those parameters are generally optimized via gradient-based search. However, gradient-based search is indeed limited to tuning parameters of a given model. Choosing the model itself is an open problem. Models are generally chosen by expert judgement, for computational convenience or by brute force search.
I present "nonparametric neural networks", a framework for jointly optimizing both parameters and model. Under this framework, we alternate between local changes to the model and gradient-based parameter tuning. The search is enabled by defining a connected, continuous space over model-parameter pairs, by penalizing models according to their complexity, and by several optimization tricks.
Presented in Partial Fulfillment of the CSD Speaking Skills Requirement.