 |
|
Jun-Yan Zhu Assistant Professor Email: junyanz at cs dot cmu dot edu |
|
|
CV | Google Scholar | GitHub | Arxiv |
|
|
|
|||||||||
| I am an Assistant Professor with The Robotics Institute in the School of Computer Science of Carnegie Mellon University. I also hold affiliated faculty appointments in the Computer Science Department and Machine Learning Department. I study computer vision, computer graphics, machine learning, and computational photography. Prior to joining CMU, I was a Research Scientist at Adobe Research. I did a postdoc at MIT CSAIL, working with William T. Freeman, Josh Tenenbaum, and Antonio Torralba. I obtained my Ph.D. from UC Berkeley, under the supervision of Alexei A. Efros. I received my B.E. from Tsinghua University, working with Zhuowen Tu, Shi-Min Hu, and Eric Chang. |
Code & News
[Notes] To prospective CMU students, if you are stressed by graduate school application, check out some Cat Papers.
[Code] PyTorch implementation and Google Colab for CycleGAN and pix2pix.
[Code] Try "pip install clean-fid". An FID calculation repo with proper image resizing and quantization steps.
[Code] Code for customizing a pre-trained GAN with one or a few hand-drawn sketches.
[Code] SDEdit: Image Synthesis and Editing with Stochastic Differential Equations.
[Code] Depth-supervised NeRF: Add depth supervision loss to your NeRF training.
[Code] Interactively editing a Conditional NeRF. Changing the color and shape of 3D regions with sparse scribbles.
[Code] GAN Ensembling: use generative models for test-time data augmentation.
[Code] Swapping Autoencoder for various image editing tasks: texture swapping, local editing, and latent code vector arithmetic.
[Code] Don't forget to use data augmentation for your GANs training. The code supports StyleGAN2-PyTorch/TF and BigGAN-PyTorch.
[Code] Anycost GAN can accelerate StyleGAN2 inference by 6-12x on diverse hardware. Try it on your laptop.
[Code] Contrastive Learning for unpaired image-to-image translation. Faster and lighter training compared to CycleGAN.
[Code] Our model rewriting code allows you to interactively edit the network weights.
[Code] for compressing pix2pix, CycleGAN, and GauGAN by 9-21x.
[Course] SIGGRAPH 2021 Course on the Advances in Neural Rendering.
[Workshop] SIGGRAPH 2021 Workshop on Measurable Creative AI.
[Tutorial] CVPR 2020 Tutorial on Neural Rendering.
[Workshop] ICCV 2019 Workshop on Image and Video Synthesis.
[CatPapers] Cool vision, learning, and graphics papers on Cats.
Research Group
| Our lab studies the connection between Data, Humans, and Generative Models, with the goal of building intelligent machines, capable of recreating our visual world and helping everyone tell their visual stories. We focus on three directions: (1) We design new generative models to help humans create visual content more easily. Our models can synthesize photorealistic outputs (e.g., images, videos, 3D data, multimodal data) given humans' simple instructions. (2) We build user interfaces and algorithms for humans to visualize, customize, and create generative models. (3) We use generative models and neural rendering to create synthetic training data for computer vision and robotics applications. |
· George Cazenavette (MSR)
· Kangle Deng (PhD, with Deva Ramanan)
· Ruihan Gao (PhD, with Wenzhen Yuan)
· Nupur Kumari (MSR)
· Muyang Li (MSR)
· Gaurav Parmar (MSR)
· Chonghyuk (Andrew) Song (MSR, with Deva Ramanan)
· Sheng-Yu Wang (PhD)
Teaching
16-726: Learning-based Image Synthesis (Spring 2022, Spring 2021): If you are on the waiting list, please contact me via email.
16-824: Visual Learning and Recognition (Fall 2021)
Deep Learning at Udacity (Co-instructor).
Software
 |
NVIDIA Canvas: Turn Simple Brushstrokes into Realistic Images Download Windows 10 app based on our work SPADE (CVPR 2019) and GauGAN demo (SIGGRAPH 2019).
|
 |
Photoshop Neural Filters Photoshop 2021 introduces "Neural Filters". Several features are partly built on our work iGAN (ECCV 2016), ideepcolor (SIGGRAPH 2017), and CycleGAN (ICCV 2017).
|
Selected Publications
See the full list on Google Scholar.
 |
Sketch Your Own GAN Sheng-Yu Wang, David Bau, Jun-Yan Zhu ICCV 2021
|
 |
SDEdit: Image Synthesis and Editing with Stochastic Differential Equations Chenlin Meng, Song Yang, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon arXiv 2108.01073, 2021
|
 |
Depth-supervised NeRF: Fewer Views and Faster Training for Free Kangle Deng, Andrew Liu, Jun-Yan Zhu, Deva Ramanan arXiv 2107.02791, 2021
|
 |
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation Gaurav Parmar, Richard Zhang, Jun-Yan Zhu arXiv 2104.11222, 2021 Installation: pip install clean-fid
|
 |
Editing Conditional Radiance Fields Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, Bryan Russell ICCV 2021
|
 |
Ensembling with Deep Generative Views Lucy Chai, Jun-Yan Zhu, Eli Shechtman, Phillip Isola, Richard Zhang CVPR 2021
|
 |
Anycost GANs for Interactive Image Synthesis and Editing Ji Lin, Richard Zhang, Frieder Ganz, Song Han, Jun-Yan Zhu CVPR 2021
|
 |
Understanding the Role of Individual Units in a Deep Neural Network David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, Antonio Torralba PNAS 2020
Project |
Code |
Paper |
Arxiv |
BibTex |
 |
Swapping Autoencoder for Deep Image Manipulation Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, and Richard Zhang NeurIPS 2020
|
 |
Differentiable Augmentation for Data-Efficient GAN Training Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, Song Han NeurIPS 2020
|
 |
Contrastive Learning for Unpaired Image-to-Image Translation Taesung Park, Alexei A. Efros, and Richard Zhang, Jun-Yan Zhu ECCV 2020
|
 |
Rewriting a Deep Generative Model David Bau, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba ECCV 2020
|
 |
The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, Antonio Torralba ECCV 2020
|
 |
Transforming and Projecting Images into Class-conditional Generative Networks Minyoung Huh, Richard Zhang, Jun-Yan Zhu, Sylvain Paris, Aaron Hertzmann ECCV 2020
|
 |
GAN Compression: Efficient Architectures for Interactive Conditional GANs Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, Song Han CVPR 2020
|
 |
Diverse Image Generation via Self-Conditioned GANs Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, Antonio Torralba CVPR 2020
|
 |
State of the Art on Neural Rendering Ayush Tewari*, Ohad Fried*, Justus Thies*, Vincent Sitzmann*, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer Eurographics 2020 (STAR Report)
Paper |
Project |
BibTex |
 |
Seeing what a GAN Cannot Generate David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, Antonio Torralba ICCV 2019
|
 |
Semantic Photo Manipulation with a Generative Image Prior David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, Antonio Torralba SIGGRAPH 2019
|
 |
Learning the Signatures of the Human Grasp Using a Scalable Tactile Glove Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik Nature, 569 (7758), 2019
|
 |
Connecting Touch and Vision via Cross-Modal Prediction Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba CVPR 2019 See CNN News
|
 |
Semantic Image Synthesis with Spatially-Adaptive Normalization Taesung Park, Ming-Yu Liu, Ting-Chun Wang, Jun-Yan Zhu CVPR 2019 (Best Paper Finalist) SIGGRAPH 2019 Real-time Live Demo "GauGAN" (with Chris Hebert and Gavriil Klimov) Won "Best in Show Award" and "Audience Choice Award" in SIGGRAPH 2019 Real-time Live.
Project |
Real-time Live |
Code |
Paper |
 |
GAN Dissection: Visualizing and Understanding Generative Adversarial Networks David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, Antonio Torralba ICLR 2019
|
 |
Propagation Networks for Model-Based Control Under Partial Observation Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, Russ Tedrake ICRA 2019
|
 |
Dataset Distillation Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros arXiv 2018
|
 |
Visual Object Networks: Image Generation with Disentangled 3D Representation Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman NeurIPS 2018
|
 |
3D-Aware Scene Manipulation via Inverse Graphics Shunyu Yao*, Tzu-Ming Harry Hsu*, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Freeman, Joshua B. Tenenbaum NeurIPS 2018
|
 |
Video-to-Video Synthesis Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro NeurIPS 2018 See our driving game demo.
|
 |
CyCADA: Cycle-Consistent Adversarial Domain Adaptation Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Alexei A. Efros, and Trevor Darrell ICML 2018
|
 |
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro
CVPR 2018 Featured in GTC 2018 Keynote.
|
 |
Spatially Transformed Adversarial Examples Chaowei Xiao*, Jun-Yan Zhu*, Bo Li, Mingyan Liu, and Dawn Song ICLR 2018
|
 |
Generating Adversarial Examples with Adversarial Networks Chaowei Xiao, Bo Li, Jun-Yan Zhu, Mingyan Liu, and Dawn Song IJCAI 2018
|
 |
Toward Multimodal Image-to-Image Translation Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A. Efros, Oliver Wang, and Eli Shechtman NeurIPS 2017
|
 |
Learning to Synthesize and Manipulate Natural Images December, 2017 ACM SIGGRAPH Outstanding Doctoral Dissertation Award. David J. Sakrison Memorial Prize for outstanding doctoral research, by the UC Berkeley EECS Dept.
|
 |
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks Jun-Yan Zhu*, Taesung Park*, Phillip Isola, and Alexei A. Efros ICCV 2017
Project |
PyTorch |
Torch |
Paper |
 |
Image-to-Image Translation with Conditional Adversarial Nets Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros CVPR 2017 See Distill blog | Also see neat uses of #pix2pix on Twitter. |
 |
Real-Time User-Guided Image Colorization with Learned Deep Priors Richard Zhang*, Jun-Yan Zhu*, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros SIGGRAPH 2017 Photoshop Element 2020 ColorizePhoto is based on our work
Project |
UI Code |
PyTorch Training |
Youtube |
Video |
 |
Light Field Video Capture Using a Learning-Based Hybrid Imaging System Ting-Chun Wang, Jun-Yan Zhu, Nima Khademi Kalantari, Alexei A. Efros, and Ravi Ramamoorthi SIGGRAPH 2017
Project |
GitHub |
Youtube |
Training code |
 |
Generative Visual Manipulation on the Natural Image Manifold Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros ECCV 2016 See Distill blog and article in California Magazine
|
 |
A 4D Light-Field Dataset and CNN Architectures for Material Recognition Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A. Efros, and Ravi Ramamoorthi ECCV 2016
Paper |
Data (thumbnail) |
Full data (15.9G) |
 |
Learning a Discriminative Model for the Perception of Realism in Composite Images Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros ICCV 2015
|
 |
Mirror Mirror: Crowdsourcing Better Portraits Jun-Yan Zhu, Aseem Agarwala, Alexei A. Efros, Eli Shechtman, and Jue Wang SIGGRAPH Asia 2014
Project (code) |
Paper |
Data |
Slides |
Supplement |
BibTex |
 |
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections Jun-Yan Zhu, Yong Jae Lee and Alexei A. Efros SIGGRAPH 2014
See article in The New Yorker |
 |
MILCut: A Sweeping Line Multiple Instance Learning Paradigm for Interactive Image Segmentation Jiajun Wu*, Yibiao Zhao*, Jun-Yan Zhu, Siwei Luo and Zhuowen Tu CVPR 2014
|
 |
Unsupervised Object Class Discovery via Saliency-Guided Multiple Class Learning Jun-Yan Zhu, Jiajun Wu, Yan Xu, Eric Chang and Zhuowen Tu TPAMI 2015 | CVPR 2012
Project |
Paper |
Supplement |
Poster |
BibTex |
 |
Multiple Clustered Instance Learning for Histopathology Cancer Image Classification, Segmentation and Clustering Yan Xu*, Jun-Yan Zhu*, Eric I-Chao Chang and Zhuowen Tu CVPR 2012 | Medical Image Analysis 2014
|
 |
Motion-Aware Gradient Domain Video Composition Tao Chen, Jun-Yan Zhu, Ariel Shamir and Shi-Min Hu TIP 2013
|
Talks
SIGGRAPH Dissertation Award Talk (2018)
Unpaired Image-to-Image Translation
CVPR Tutorial on GANs (2018)
Learning to Synthesize and Manipulate Natural Photos
MIT, HKUST CSE Departmental Seminar, ICCV Tutorial on GANs, O'Reilly AI, AI with the best, Y Conf, DEVIEW, ODSC West (2017)
Stanford, MIT, Facebook, CUHK, SNU (2017)
SIGGRAPH, NVIDIA Innovation Theater, Global AI Hackathon (2017)
Visual Manipulation and Synthesis on the Natural Image Manifold
Facebook, MSR, Berkeley BAIR, THU, ICML workshop "Visualization for Deep Learning" (2016)
Mirror Mirror: Crowdsourcing Better Portraits
SIGGRAPH Asia (2014)
What Makes Big Visual Data Hard?
SIGGRAPH Asia invited course "Data-Driven Visual Computing" (2014)
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections
SIGGRAPH (2014)
Events
[Course] SIGGRAPH 2021 Course on the Advances in Neural Rendering
[Workshop] SIGGRAPH 2021 Workshop on Measurable Creative AI
[Workshop] CVPR 2021 Workshop on Computational Measurements of Machine Creativity
[Tutorial] CVPR 2020 Tutorial on Neural Rendering
[Tutorial] Eurographics 2020 STAR on Neural Rendering
[Journal] IJCV Special Issue on Generative Adversarial Networks for Computer Vision (2019-2020)
[Workshop] ICCV 2019 Workshop on Image and Video Synthesis.
[Tutorial] CVPR 2019 Tutorial on Map Synchronization.
[Tutorial] CVPR 2018 Tutorial on Generative Adversarial Networks.
[Tutorial] ICCV 2017 Tutorial on Generative Adversarial Networks.
[Workshop] ICML 2017 Workshop on Visualization for Deep Learning.
[Course] SIGGRAPH Asia 2014 invited Course on Data-Driven Visual Computing.
MISC
· My cat Aquarius and my dog Arya's photo and its Ukiyo-e style.
· You can eat Greek yogurt when wearing a mask. See how Arya does it.
