Language Technologies Thesis Proposal
- Gates Hillman Centers
- HIEU PHAM
- Ph.D. Student
- Language Technologies Institute
- Carnegie Mellon University
Efficient Machine Learning
I study the methods that make deep learning algorithms more efficient in three facets: 1) model training time; 2) model designing time; and 3) data complexity.
Next, in Chapter 2, I consider the problem of device placement for neural networks. Given a network, e.g. InceptionV3 and a list ofavailable devices (e.g. 4 GPUs and 1 CPU), and one has to decide which parts of the given network should run on which device to minimize the network's execution time. I treat the task as a combinatorial optimization problem and apply NCO. NCO finds placements that are up to 23\% faster than the placements designed by human experts.
Then, in Chapter 3, I consider the problem of designing neural network architectures. This task is known as Neural Architecture Search (NAS), and can be posed as a discrete optimization problem, but with the bottleneck that evaluating reward functions is very expensive. I apply NCO on this task, with a significant motivation where all architectures are forced to share their parameters. The resulting algorithms, Efficient Neural Architecture Search (ENAS) could achieve similar performance to NAS, but is 1,000x to 50,000x more efficient.
Then, in Chapter 4, I apply ENAS to design architecture for Neural Machine Translation (NMT). I found that the search space is crucial for ENAS. The search space I designed for language model (Chapter 3) appears not compatible with NMT. An alternate search space is being designed and my preliminary experiments show that such alternate space may work well.
Finally, in Chapter 5, I design an unsupervised learning algorithm to reduce the amount of data needed (and hence, also improve the training time needed) for image classification.
Jaime Carbonell (Co-advisor)
Quoc V. Le (Co-advisor) (Google Brain)
Samy Bengio (Google Brain)
Chris Dyer (CMU and /DeepMind)
Copy of Thesis Proposal