Machine Learning Thesis Defense

  • Ph.D. Student
  • Machine Learning Department
  • Carnegie Mellon University
Thesis Orals

One of the fundamental problems in machine learning and statistics is learn- ing generative models of data. Explicit generative models, which model probability densities of data, have been intensively studied in numerous applications. However, using combinations of simple parametric distributions is usually difficult to model complex data, such as natural images. Implicit generative models (IGMs), which model transformations between known source distributions and target distributions to simulate the sampling process without specifying densities explicitly, regain attention with explosion of interests. With recent success of deep learning, IGMs have yielded impressive empirical performance in different applications.

While there are new algorithms for learning IGMs, its theoretical proper- ties are weakly justified and the links with existing methods is under explored. The first thrust of this thesis is to understand statistical guarantees of learning IGMs. By connecting IGMs with two-sample test, we propose a new generic algorithm that can be built on the top of many existing approaches and bring performance improvement over the state-of-the-art. On the other hand, from the perspective of statistical analysis, IGMs, which model transformations, is fundamentally different from traditional explicit models, which makes the existing results not directly applicable. We then study error bounds and sample complexities of learning IGMs, which makes a step forward in building its rigorous foundations.

In the second part, we shift our focus to different types of data that we are interested in. We develop algorithms for learning IGMs on various data, which range from images, text, and point clouds, by exploiting their underlying structures. Instead of modeling IGM transformations blindly via powerful functions only, such as deep neural networks, we propose to leverage human priors into model designs to reduce model sizes, save computational overhead, and have interpretable results. In this thesis, we show an example of incorporating a simple yet representative enough renderer developed in computer graphics into IGMs for generating realistic and highly structured body data, which opens a new route of learning IGMs.

Finally, we study how IGMs can improve existing machine learning algorithms. By thinking IGMs as learning transformations, we could cast target mappings in numerous tasks as IGM transformations. This perspective brings broader applications to IGMs from different domains. We present an example of how to learn proximal operators as IGM transformations to solve important linear inverse problems in computer vision. In the second example, we introduce a new way of using IGMs by treating them as auxiliary components to benefit non-generative tasks while the final output of the interest is not the generative models. We present an application of optimizing test power in anomaly detection, by constructing a power lower bound via auxiliary IGMs. 

Thesis Committee:
Barnabás Póczos (Chair)
Jeff Schneider
Ruslan Salakhutdinov
Phillip Isola (MIT)

Additional Thesis Information

For More Information, Please Contact: