|
|
Jun-Yan Zhu Assistant Professor Email: junyanz at cs dot cmu dot edu |
|
|
Students | CV | Google Scholar | GitHub | Arxiv |
|
|
|
|||||||||
|
I am an Assistant Professor with The Robotics Institute in the School of Computer Science of Carnegie Mellon University. I also hold affiliated faculty appointments in the Computer Science Department and Machine Learning Department. I study computer vision, computer graphics, machine learning, and computational photography. Prior to joining CMU, I was a Research Scientist at Adobe Research. I did a postdoc at MIT CSAIL, working with William T. Freeman, Josh Tenenbaum, and Antonio Torralba. I obtained my Ph.D. from UC Berkeley, under the supervision of Alexei A. Efros. I received my B.E. from Tsinghua University, working with Zhuowen Tu, Shi-Min Hu, and Eric Chang. |
Code & Events
Generative Intelligence Lab
|
Our lab studies the connection between Data, Humans, and Generative Models, with the goal of building intelligent machines, capable of recreating our visual world and helping everyone tell their visual stories. We focus on three directions: (1) We design generative models to help humans create visual data effortlessly. Our models can synthesize realistic outputs (e.g., images, videos, 3D data, multimodal data) given humans' simple instructions. (2) We build user interfaces and algorithms for humans to visualize, create, share, and discover generative models more easily. (3) We automatically create synthetic data for training computer vision and robotics systems. Our lab is part of Carnegie Mellon Graphics Lab and Carnegie Mellon Computer Vision Group. |
|
Former members and visitors: Rohan Agarwal (MSCV, now at Runway ML), George Cazenavette (MSR, now PhD student at MIT), Bingliang Zhang (Undergrad) |
Teaching
Software
 |
Landscape Mixer Photoshop 2022's Landscape Mixer can transform landscape images in various ways. This feature is based on our work Swapping Autoencoder (NeurIPS 2020).
|
 |
NVIDIA Canvas: Turn Simple Brushstrokes into Realistic Images Download Windows 10 app based on our work SPADE (CVPR 2019) and GauGAN demo (SIGGRAPH 2019).
|
 |
Photoshop Neural Filters Photoshop 2021 introduces "Neural Filters". Several features are partly built on our work iGAN (ECCV 2016), ideepcolor (SIGGRAPH 2017), and CycleGAN (ICCV 2017).
|
Selected Publications
See the full list on Google Scholar.
 |
Content-Based Search for Deep Generative Models Daohan Lu*, Sheng-Yu Wang*, Nupur Kumari*, Rohan Agarwal*, David Bau, Jun-Yan Zhu arXiv 2022
|
 |
Multi-Concept Customization of Text-to-Image Diffusion Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, Jun-Yan Zhu arXiv 2022
|
 |
Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, Jun-Yan Zhu NeurIPS 2022
|
 |
Rewriting Geometric Rules of a GAN Sheng-Yu Wang, David Bau, Jun-Yan Zhu SIGGRAPH 2022
|
|
GAN-Supervised Dense Visual Alignment William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A. Efros, Eli Shechtman CVPR 2022 (Best Paper Finalist)
|
 |
Ensembling Off-the-shelf Models for GAN Training Nupur Kumari, Richard Zhang, Eli Shechtman, Jun-Yan Zhu CVPR 2022 Installation: pip install vision-aided-loss
|
 |
On Aliased Resizing and Surprising Subtleties in GAN Evaluation Gaurav Parmar, Richard Zhang, Jun-Yan Zhu CVPR 2022 Installation: pip install clean-fid
|
 |
Dataset Distillation by Matching Training Trajectories George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, Jun-Yan Zhu CVPR 2022
|
 |
Depth-supervised NeRF: Fewer Views and Faster Training for Free Kangle Deng, Andrew Liu, Jun-Yan Zhu, Deva Ramanan CVPR 2022
|
 |
Spatially-Adaptive Multilayer Selection for GAN Inversion and Editing Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, Krishna Kumar Singh CVPR 2022
|
 |
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations Chenlin Meng, Yutong He, Song Yang, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon ICLR 2022
|
 |
Sketch Your Own GAN Sheng-Yu Wang, David Bau, Jun-Yan Zhu ICCV 2021
|
 |
Editing Conditional Radiance Fields Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan Russell ICCV 2021
|
 |
Ensembling with Deep Generative Views Lucy Chai, Jun-Yan Zhu, Eli Shechtman, Phillip Isola, Richard Zhang CVPR 2021
|
 |
Anycost GANs for Interactive Image Synthesis and Editing Ji Lin, Richard Zhang, Frieder Ganz, Song Han, Jun-Yan Zhu CVPR 2021
|
 |
GAN Compression: Efficient Architectures for Interactive Conditional GANs Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, Song Han TPAMI 2021 | CVPR 2020
|
 |
Understanding the Role of Individual Units in a Deep Neural Network David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, Antonio Torralba PNAS 2020
Project |
Code |
Paper |
Arxiv |
BibTex |
 |
Swapping Autoencoder for Deep Image Manipulation Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang NeurIPS 2020
|
 |
Differentiable Augmentation for Data-Efficient GAN Training Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, Song Han NeurIPS 2020
|
 |
Contrastive Learning for Unpaired Image-to-Image Translation Taesung Park, Alexei A. Efros, and Richard Zhang, Jun-Yan Zhu ECCV 2020
|
 |
Rewriting a Deep Generative Model David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba ECCV 2020
|
 |
The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, Antonio Torralba ECCV 2020
|
 |
Transforming and Projecting Images into Class-conditional Generative Networks Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, Aaron Hertzmann ECCV 2020
|
 |
Diverse Image Generation via Self-Conditioned GANs Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, Antonio Torralba CVPR 2020
|
 |
State of the Art on Neural Rendering Ayush Tewari*, Ohad Fried*, Justus Thies*, Vincent Sitzmann*, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer Eurographics 2020 (STAR Report)
Paper |
Project |
BibTex |
 |
Seeing what a GAN Cannot Generate David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba ICCV 2019
|
 |
Semantic Photo Manipulation with a Generative Image Prior David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba SIGGRAPH 2019
|
 |
Learning the Signatures of the Human Grasp Using a Scalable Tactile Glove Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik Nature, 569 (7758), 2019
|
 |
Connecting Touch and Vision via Cross-Modal Prediction Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba CVPR 2019 See CNN News
|
 |
Semantic Image Synthesis with Spatially-Adaptive Normalization Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu CVPR 2019 (Best Paper Finalist) SIGGRAPH 2019 Real-time Live Demo "GauGAN" (with Chris Hebert and Gavriil Klimov) Won "Best in Show Award" and "Audience Choice Award" in SIGGRAPH 2019 Real-time Live.
Project |
Real-time Live |
Code |
Paper |
 |
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba ICLR 2019
|
 |
Propagation Networks for Model-Based Control Under Partial Observation Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, Russ Tedrake ICRA 2019
|
 |
Dataset Distillation Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros arXiv 2018
|
 |
Visual Object Networks: Image Generation with Disentangled 3D Representation Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman NeurIPS 2018
|
 |
3D-Aware Scene Manipulation via Inverse Graphics Shunyu Yao*, Tzu-Ming Harry Hsu*, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, Joshua B. Tenenbaum NeurIPS 2018
|
 |
Video-to-Video Synthesis Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro NeurIPS 2018 See our driving game demo.
|
 |
CyCADA: Cycle-Consistent Adversarial Domain Adaptation Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Alexei A. Efros, and Trevor Darrell ICML 2018
|
 |
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro
CVPR 2018 Featured in GTC 2018 Keynote.
|
 |
Spatially Transformed Adversarial Examples Chaowei Xiao*, Jun-Yan Zhu*, Bo Li, Mingyan Liu, and Dawn Song ICLR 2018
|
 |
Generating Adversarial Examples with Adversarial Networks Chaowei Xiao, Bo Li, Jun-Yan Zhu, Mingyan Liu, and Dawn Song IJCAI 2018
|
 |
Toward Multimodal Image-to-Image Translation Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman NeurIPS 2017
|
 |
Learning to Synthesize and Manipulate Natural Images December, 2017 ACM SIGGRAPH Outstanding Doctoral Dissertation Award. David J. Sakrison Memorial Prize for outstanding doctoral research, by the UC Berkeley EECS Dept.
|
 |
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros ICCV 2017
Project |
PyTorch |
Torch |
Paper |
 |
Image-to-Image Translation with Conditional Adversarial Nets Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros CVPR 2017 See Distill blog | Also see neat uses of #pix2pix on Twitter. |
 |
Real-Time User-Guided Image Colorization with Learned Deep Priors Richard Zhang*, Jun-Yan Zhu*, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros SIGGRAPH 2017 Photoshop Element 2020 ColorizePhoto is based on our work
Project |
UI Code |
PyTorch Training |
Youtube |
Video |
 |
Light Field Video Capture Using a Learning-Based Hybrid Imaging System Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi SIGGRAPH 2017
Project |
GitHub |
Youtube |
Training code |
 |
Generative Visual Manipulation on the Natural Image Manifold Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros ECCV 2016 See Distill blog and article in California Magazine
|
 |
A 4D Light-Field Dataset and CNN Architectures for Material Recognition Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi ECCV 2016
Paper |
Data (thumbnail) |
Full data (15.9G) |
 |
Learning a Discriminative Model for the Perception of Realism in Composite Images Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros ICCV 2015
|
 |
Mirror Mirror: Crowdsourcing Better Portraits Jun-Yan Zhu, Aseem Agarwala, Alexei A. Efros, Eli Shechtman, and Jue Wang SIGGRAPH Asia 2014
Project (code) |
Paper |
Data |
Slides |
Supplement |
BibTex |
 |
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections Jun-Yan Zhu, Yong Jae Lee and Alexei A. Efros SIGGRAPH 2014
See article in The New Yorker |
 |
MILCut: A Sweeping Line Multiple Instance Learning Paradigm for Interactive Image Segmentation Jiajun Wu*, Yibiao Zhao*, Jun-Yan Zhu, Siwei Luo and Zhuowen Tu CVPR 2014
|
 |
Unsupervised Object Class Discovery via Saliency-Guided Multiple Class Learning Jun-Yan Zhu, Jiajun Wu, Yan Xu, Eric Chang and Zhuowen Tu TPAMI 2015 | CVPR 2012
Project |
Paper |
Supplement |
Poster |
BibTex |
 |
Multiple Clustered Instance Learning for Histopathology Cancer Image Classification, Segmentation and Clustering Yan Xu*, Jun-Yan Zhu*, Eric I-Chao Chang and Zhuowen Tu CVPR 2012 | Medical Image Analysis 2014
|
 |
Motion-Aware Gradient Domain Video Composition Tao Chen, Jun-Yan Zhu, Ariel Shamir and Shi-Min Hu TIP 2013
|
Talks
SIGGRAPH Dissertation Award Talk (2018)
Unpaired Image-to-Image Translation
CVPR Tutorial on GANs (2018)
Learning to Synthesize and Manipulate Natural Photos
MIT, HKUST CSE Departmental Seminar, ICCV Tutorial on GANs, O'Reilly AI, AI with the best, Y Conf, DEVIEW, ODSC West (2017)
Stanford, MIT, Facebook, CUHK, SNU (2017)
SIGGRAPH, NVIDIA Innovation Theater, Global AI Hackathon (2017)
Visual Manipulation and Synthesis on the Natural Image Manifold
Facebook, MSR, Berkeley BAIR, THU, ICML workshop "Visualization for Deep Learning" (2016)
Mirror Mirror: Crowdsourcing Better Portraits
SIGGRAPH Asia (2014)
What Makes Big Visual Data Hard?
SIGGRAPH Asia invited course "Data-Driven Visual Computing" (2014)
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections
SIGGRAPH (2014)
Past Events
MISC
