Spatially-Adaptive Multilayer Selection for GAN Inversion and Editing
Gaurav Parmar 1,2
Yijun Li 2
Jingwan (Cynthia) Lu 2
Richard Zhang 2
Jun-Yan Zhu 1
Krishna Kumar Singh 2
1 Carnegie Mellon University
2Adobe Research

CVPR 2022

[Paper]
[GitHub]

Choosing a single latent layer for GAN inversion leads to a dilemma between obtaining a faithful reconstruction of the input image and being able to perform downstream edits (1st and 2nd row). In contrast, our proposed method automatically selects the latent space tailored for each region to balance the reconstruction quality and editability (3rd row).


Abstract

Existing GAN inversion and editing methods work well for aligned objects with a clean background, such as portraits and animal faces, but often struggle for more difficult categories with complex scene layouts and object occlusions, such as cars, animals, and outdoor images. We propose a new method to invert and edit such complex images in the latent space of GANs, such as StyleGAN2. Our key idea is to explore inversion with a collection of layers, spatially adapting the inversion process to the difficulty of the image. We learn to predict the "invertibility" of different image segments and project each segment into a latent layer. Easier regions can be inverted into an earlier layer in the generator's latent space, while more challenging regions can be inverted into a later feature space. Experiments show that our method obtains better inversion results compared to the recent approaches on complex categories, while maintaining downstream editability.


Different Latent Spaces

W+
F4
F6
F8
F10
SAM

Results

Cars

Input W+ Inversion SAM Inversion
Better
Editing
Better
Inversion

Invertibility Map Used

W+

F4

F6

F8

F10

Edit Name #1: "Blue car"
Edit Name #2: "Ford Model T"

Paper

Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, Krishna Kumar Singh
Spatially-Adaptive Multilayer Selection for GAN Inversion and Editing
CVPR, 2022.

[Bibtex]

Acknowledgements

We thank Eli Shechtman, Sheng-Yu Wang, Nupur Kumari, Kangle Deng, George Cazenavette, Ruihan Gao, and Chonghyuk (Andrew) Song for useful discussions. We are grateful for the support from Adobe, Naver Corporation, and Sony Corporation. This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.