Multi-Concept Customization of Text-to-Image Diffusion

Nupur Kumari¹

Bingliang Zhang²

Richard Zhang³

Eli Shechtman³

Jun-Yan Zhu¹

¹ CMU ²Tsinghua University ³Adobe Research

CVPR 2023

Code Paper Project Gallery Slides Data

We release a dataset consisting of 101 concepts with 3-15 images in each concept for evaluating model customization methods. Target real images of each concept in the Dataset are shown below.

We introduce both single-concept and multi-concept settings with evaluation text prompts for each case. Below we show random samples with Ours, DreamBooth, and Textual Inversion method for each concept. Scroll horizontally to see all samples with different test prompts.

Dataset and prompt creation: we collected images from Unsplash or ourselves for concepts across a variety of categories, namely, toys, plushies, wearables, scenes, transport vehicles, furniture, home decor items, luggage, human faces, musical instruments, rare flowers, food items, pet animals. For creating evaluation prompts, we first used ChatGPT to generate 40 image captions for each concept with the instructions to either (1) change the background while keeping the main subject, (2) insert a new object/living thing in the scene along with the main subject, (3) style variation of the main subject, and (4) change the property or material of the main subject. The generated text prompts are manually filtered or modified to get the final 20 prompts for each concept. A similar strategy is applied for multiple concepts. Some of the prompts are also inspired by other concurrent works e.g. Perfusion, DreamBooth, SuTI, BLIP-Diffusion etc.

License: Images taken from UnSplash are under Unsplash License. Images collected by us are released under CC BY-SA 4.0 license. Flower category images are downloaded from Wikimedia/Flickr/Pixabay and the link to orginial images can also be found here

Please refer to our code for details regarding dataset download, text prompts, and evaluation code for single-concept and multi-concept customization.

Action Figure

Custom Diffusion

Dreambooth

Textual Inversion

Action Figure

Custom Diffusion

Dreambooth

Textual Inversion

Figurine

Custom Diffusion

Dreambooth

Textual Inversion

Houseplant

Custom Diffusion

Dreambooth

Textual Inversion

Houseplant

Custom Diffusion

Dreambooth

Textual Inversion

Houseplant

Custom Diffusion

Dreambooth

Textual Inversion

Lamp

Custom Diffusion

Dreambooth

Textual Inversion

Vase

Custom Diffusion

Dreambooth

Textual Inversion

Wooden Pot

Custom Diffusion

Dreambooth

Textual Inversion

Dish

Custom Diffusion

Dreambooth

Textual Inversion

Dish

Custom Diffusion

Dreambooth

Textual Inversion

Flower

Custom Diffusion

Dreambooth

Textual Inversion

Flower

Custom Diffusion

Dreambooth

Textual Inversion

Chair

Custom Diffusion

Dreambooth

Textual Inversion

Chair

Custom Diffusion

Dreambooth

Textual Inversion

Chair

Custom Diffusion

Dreambooth

Textual Inversion

Sofa

Custom Diffusion

Dreambooth

Textual Inversion

Sofa

Custom Diffusion

Dreambooth

Textual Inversion

Table

Custom Diffusion

Dreambooth

Textual Inversion

Guitar Amplifier

Custom Diffusion

Dreambooth

Textual Inversion

Guitar

Custom Diffusion

Dreambooth

Textual Inversion

Guitar

Custom Diffusion

Dreambooth

Textual Inversion

Violin

Custom Diffusion

Dreambooth

Textual Inversion

Earrings

Custom Diffusion

Dreambooth

Textual Inversion

Ring

Custom Diffusion

Dreambooth

Textual Inversion

Backpack

Custom Diffusion

Dreambooth

Textual Inversion

Purse

Custom Diffusion

Dreambooth

Textual Inversion

Purse

Custom Diffusion

Dreambooth

Textual Inversion

Purse

Custom Diffusion

Dreambooth

Textual Inversion

Purse

Custom Diffusion

Dreambooth

Textual Inversion

Jun-Yan Zhu

Custom Diffusion

Dreambooth

Textual Inversion

Richard Zhang

Custom Diffusion

Dreambooth

Textual Inversion

Eli Shechtman

Custom Diffusion

Dreambooth

Textual Inversion

Cat

Custom Diffusion

Dreambooth

Textual Inversion

Cat

Custom Diffusion

Dreambooth

Textual Inversion

Cat

Custom Diffusion

Dreambooth

Textual Inversion

Cat

Custom Diffusion

Dreambooth

Textual Inversion

Cat

Custom Diffusion

Dreambooth

Textual Inversion

Cat

Custom Diffusion

Dreambooth

Textual Inversion

Cat

Custom Diffusion

Dreambooth

Textual Inversion

Dog

Custom Diffusion

Dreambooth

Textual Inversion

Dog

Custom Diffusion

Dreambooth

Textual Inversion

Dog

Custom Diffusion

Dreambooth

Textual Inversion

Dog

Custom Diffusion

Dreambooth

Textual Inversion

Pokemon Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Bunny Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Cow Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Dice Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Lobster Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Panda Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Penguin Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Teddybear

Custom Diffusion

Dreambooth

Textual Inversion

Tortoise Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Unicorn Plushie

Custom Diffusion

Dreambooth

Textual Inversion

Barn

Custom Diffusion

Dreambooth

Textual Inversion

Canal Scene

Custom Diffusion

Dreambooth

Textual Inversion

Castle

Custom Diffusion

Dreambooth

Textual Inversion

Garden

Custom Diffusion

Dreambooth

Textual Inversion

Lighthouse

Custom Diffusion

Dreambooth

Textual Inversion

Sculpture

Custom Diffusion

Dreambooth

Textual Inversion

Waterfall

Custom Diffusion

Dreambooth

Textual Inversion

Book

Custom Diffusion

Dreambooth

Textual Inversion

Book

Custom Diffusion

Dreambooth

Textual Inversion

Bottle

Custom Diffusion

Dreambooth

Textual Inversion

Corkscrew

Custom Diffusion

Dreambooth

Textual Inversion

Cup

Custom Diffusion

Dreambooth

Textual Inversion

Cup

Custom Diffusion

Dreambooth

Textual Inversion

Cup

Custom Diffusion

Dreambooth

Textual Inversion

Headphone

Custom Diffusion

Dreambooth

Textual Inversion

Headphone

Custom Diffusion

Dreambooth

Textual Inversion

Helmet

Custom Diffusion

Dreambooth

Textual Inversion

Keychain

Custom Diffusion

Dreambooth

Textual Inversion

Bear

Custom Diffusion

Dreambooth

Textual Inversion

Toy Gnome

Custom Diffusion

Dreambooth

Textual Inversion

Pokemon Toy

Custom Diffusion

Dreambooth

Textual Inversion

Kids Table Chair

Custom Diffusion

Dreambooth

Textual Inversion

Toy

Custom Diffusion

Dreambooth

Textual Inversion

Bike

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Car

Custom Diffusion

Dreambooth

Textual Inversion

Motorbike

Custom Diffusion

Dreambooth

Textual Inversion

Tank

Custom Diffusion

Dreambooth

Textual Inversion

Glasses

Custom Diffusion

Dreambooth

Textual Inversion

Jacket

Custom Diffusion

Dreambooth

Textual Inversion

Jacket

Custom Diffusion

Dreambooth

Textual Inversion

Shoes

Custom Diffusion

Dreambooth

Textual Inversion

Shoes

Custom Diffusion

Dreambooth

Textual Inversion

Sunglasses

Custom Diffusion

Dreambooth

Textual Inversion

Sunglasses

Custom Diffusion

Dreambooth

Textual Inversion

Acknowledgements

We are grateful to Sheng-Yu Wang, Songwei Ge, Daohan Lu, Ruihan Gao, Roni Shechtman, Avani Sethi, Yijia Wang, Shagun Uppal, and Zhizhuo Zhou for helping with the dataset collection, and Nick Kolkin for the feedback.