Multi-Concept Customization of Text-to-Image Diffusion

1 CMU    2Tsinghua University    3Adobe Research

CVPR 2023

Code Paper Project Gallery Slides Data

Single-Concept Results and Comparisons

We show randomly selected generated samples of our method (2nd column), DreamBooth (3rd column), and Textual Inversion (4rth column). DreamBooth fine-tunes all the parameters in the diffusion model, keeping the text transformer frozen, and uses generated images as the regularization dataset. Textual Inversion only optimizes a new word embedding token for each concept.

Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* dog


Target Images

V* dog swimming in a pool


Target Images

V* dog at a beach with a view of seashore


Target Images

V* dog in times square


Target Images

V* dog wearing sunglasses


Target Images

V* dog in a construction outfit


Target Images

V* dog playing with a ball


Target Images

V* dog wearing headphones


Target Images

V* dog oil painting ghibli inspired


Target Images

Painting of V* dog at a beach by artist claude monet


Target Images

V* dog digital painting 3d render geometric style


Target Images

Georgia O'Keeffe style V* dog painting


Target Images

A screaming V* dog


Target Images

A barking V* dog


Target Images

A depressed V* dog


Target Images

A sleeping V* dog


Target Images

A sad V* dog


Target Images

A joyous V* dog


Target Images

A frowning V* dog


Target Images

A sculpture of V* dog


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of a moongate


Target Images

Moongate in snowy ice


Target Images

Moongate at a beach with a view of seashore


Target Images

Moongate in the middle of highway road


Target Images

Moongate in times square


Target Images

Moongate on the surface of moon


Target Images

Puppy in front of moongate close up view


Target Images

Cat sitting in front of moongate


Target Images

Cat sitting in front of moongate in snowy ice


Target Images

Squirrel in front of moongate


Target Images

Moongate oil painting ghibli inspired


Target Images

Moongate painting by artist claude monet


Target Images

Moongate digital painting 3d render geometric style


Target Images

Georgia O'Keeffe style moongate painting


Target Images

A watercolor painting of moongate in a forest


Target Images

A transparent moongate made with glass


Target Images

Moongate made of metal


Target Images

Moongate in rainbow colors


Target Images

Moongate sculpture


Target Images

Moongate entrance into a medieval castle


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* cat


Target Images

V* cat swimming in a pool


Target Images

V* cat at a beach with a view of seashore


Target Images

V* cat sitting on a window


Target Images

V* cat in times square


Target Images

V* cat wearing sunglasses


Target Images

Construction outfit wearing V* cat


Target Images

V* cat playing with a ball


Target Images

V* cat wearing headphones


Target Images

V* cat oil painting ghibli inspired


Target Images

Painting of V* cat at a beach by artist claude monet


Target Images

V* cat digital painting 3d render geometric style


Target Images

Georgia O'Keeffe style V* cat painting


Target Images

A screaming V* cat


Target Images

A depressed V* cat


Target Images

A sleeping V* cat


Target Images

A sad V* cat


Target Images

An angry V* cat


Target Images

A crying V* cat


Target Images

A sculpture of V* cat


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* table


Target Images

V* table near a pool


Target Images

V* table at a beach with a view of seashore


Target Images

V* table in a garden


Target Images

V* table in grand canyon


Target Images

V* table floating in ocean


Target Images

V* table and an armchair


Target Images

A maple tree on the side of V* table


Target Images

V* table and an orange sofa


Target Images

V* table with chocolate cake on it


Target Images

V* table with a vase of rose flowers on it


Target Images

V* table oil painting ghibli inspired


Target Images

V* table painting by artist claude monet


Target Images

A digital illustration of V* table


Target Images

Georgia O'Keeffe style V* table painting


Target Images

A watercolor painting of V* table on a beach


Target Images

An orange V* table


Target Images

A pink V* table


Target Images

A red color V* table


Target Images

A study V* table in the style of V* table


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* chair


Target Images

V* chair near a pool


Target Images

V* chair at a beach with a view of seashore


Target Images

V* chair in a garden


Target Images

V* chair in grand canyon


Target Images

V* chair in front of a medieval castle


Target Images

V* chair and a coffee table


Target Images

Floor lamp on the side of V* chair


Target Images

V* chair and an orange sofa


Target Images

V* chair and a table with chocolate cake on it


Target Images

V* chair oil painting ghibli inspired


Target Images

V* chair painting by artist claude monet


Target Images

A watercolor painting of V* chair in a forest


Target Images

A digital illustration of V* chair


Target Images

Georgia O'Keeffe style V* chair painting


Target Images

An orange V* chair


Target Images

A pink V* chair


Target Images

A red color V* chair


Target Images

V* chair crochet


Target Images

An egg chair in the style of V* chair


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* tortoise plushy


Target Images

V* tortoise plushy in grand canyon


Target Images

V* tortoise plushy swimming in pool


Target Images

V* tortoise plushy sitting at the beach with a view of sea


Target Images

V* tortoise plushy in times square


Target Images

V* tortoise plushy in front of a medieval castle


Target Images

V* tortoise plushy wearing sunglasses


Target Images

V* tortoise plushy working on the laptop


Target Images

V* tortoise plushy on a boat in the sea


Target Images

V* tortoise plushy wearing headphones


Target Images

V* tortoise plushy oil painting ghibli inspired


Target Images

V* tortoise plushy painting by artist claude monet


Target Images

A digital illustration of V* tortoise plushy


Target Images

Georgia O'Keeffe style V* tortoise plushy painting


Target Images

V* tortoise plushy like tile mosaic


Target Images

A watercolor painting of V* tortoise plushy on a mountain


Target Images

A koala in the style of V* tortoise plushy


Target Images

A backpack in the style of V* tortoise plushy


Target Images

V* tortoise plushy made of crochet


Target Images

A sweater in the style of V* tortoise plushy


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of a V* barn


Target Images

V* barn in snowy ice


Target Images

V* barn in fall season with leaves all around


Target Images

V* barn at a beach with a view of seashore


Target Images

Photo of V* barn with sun rising in the sky


Target Images

puppy in front of V* barn


Target Images

Cat sitting in front of V* barn


Target Images

Cat sitting in front of V* barn in snowy ice


Target Images

Squirrel in front of V* barn


Target Images

V* barn oil painting ghibli inspired


Target Images

V* barn painting by artist claude monet


Target Images

V* barn digital painting 3d render geometric style


Target Images

Georgia O'Keeffe style V* barn painting


Target Images

a watercolor painting of V* barn


Target Images

A futuristic V* barn


Target Images

A surreal landscape V* barn


Target Images

A close up shot of V* barn


Target Images

Top view of V* barn


Target Images

V* barn with forest in the background


Target Images

Painting of V* barn in the style of van gogh


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* flower


Target Images

V* flower growing in desert


Target Images

V* flower at a beach with a view of seashore


Target Images

V* flower on top of a mountain with surise in background


Target Images

A bouquet of V* flower


Target Images

A garden of V* flower


Target Images

V* flower in a violet vase on a table


Target Images

A vase filled with V* flower on a table


Target Images

Bouquet of V* flower and roses


Target Images

V* flower and a chocolate cake on the table


Target Images

V* flower oil painting ghibli inspired


Target Images

V* flower painting by artist claude monet


Target Images

A digital illustration of V* flower


Target Images

Georgia O'Keeffe style V* flower painting


Target Images

A watercolor painting of V* flower and a teapot on the table


Target Images

V* flower with violet color petals


Target Images

A sky blue color V* flower


Target Images

A red color V* flower


Target Images

Dried up V* flower


Target Images

A dahlia flower in the style of V* flower


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* teddybear


Target Images

V* teddybear in grand canyon


Target Images

V* teddybear swimming in pool


Target Images

V* teddybear sitting at the beach with a view of sea


Target Images

V* teddybear in times square


Target Images

V* teddybear wearing sunglasses


Target Images

V* teddybear working on the laptop


Target Images

V* teddybear on a boat in the sea


Target Images

V* teddybear wearing headphones


Target Images

V* teddybear in a construction outfit


Target Images

V* teddybear oil painting ghibli inspired


Target Images

V* teddybear painting by artist claude monet


Target Images

A digital illustration of V* teddybear


Target Images

Georgia O'Keeffe style V* teddybear painting


Target Images

A watercolor painting of V* teddybear on top of a mountain


Target Images

A koala in the style of V* teddybear


Target Images

A backpack in the style of V* teddybear


Target Images

V* teddybear made of crochet


Target Images

A blanket in the style of V* teddybear


Target Images

A screaming V* teddybear


Custom Diffusion (Ours)

DreamBooth

Textual Inversion

Target Images

Photo of V* wooden pot


Target Images

V* wooden pot in grand canyon


Target Images

V* wooden pot with mountains and sunset in background


Target Images

V* wooden pot floating in a pool


Target Images

A wide shot of V* wooden pot in times square


Target Images

V* wooden pot and chocolate cake on a table


Target Images

Rose flowers in V* wooden pot on a table


Target Images

Marigold flowers in V* wooden pot


Target Images

V* wooden pot at the entrance into a medieval castle


Target Images

V* wooden pot with pens in it


Target Images

V* wooden pot oil painting ghibli inspired


Target Images

V* wooden pot painting by artist claude monet


Target Images

A watercolor painting of V* wooden pot


Target Images

A digital illustration of V* wooden pot


Target Images

Georgia O'Keeffe style V* wooden pot painting


Target Images

A handbag in the style of V* wooden pot


Target Images

a teapot in the style of V* wooden pot


Target Images

V* wooden pot made of stone


Target Images

A gate in the style of V* wooden pot


Target Images

A coffee cup in the style of V* wooden pot on a table



Multi-Concept Results and Comparisons

We show randomly generated samples of our joint training (2nd column), optimization based (3rd column) approach, and DreamBooth (4rth column) for multi-concept composition.

Ours (Joint)

Ours (Optimization)

DreamBooth

Target Images

A V2* chair and V1* cat sitting on it


Target Images

A V2* chair and a V1* cat


Target Images

V2* chair with a screaming V1* cat sitting on it


Target Images

V2* chair with the V1* cat sitting on it near a beach


Target Images

V2* chair with the V1* cat sitting near a pool


Target Images

Watercolor painting of V1* cat sitting on V2* chair


Target Images

A digital illustration of V1* cat sitting on V2* chair


Target Images

Mountains in the background and V1* cat is sitting on V2* chair


Ours (Joint)

Ours (Optimization)

DreamBooth

Target Images

V2* dog in front of moongate


Target Images

Moongate and V2* dog


Target Images

V2* dog wearing sunglasses in front of moongate


Target Images

V2* dog in front of moongate in snowy ice


Target Images

Watercolor painting of the V2* dog in front of moongate


Target Images

A digital illustration of the V2* dog in front of moongate


Target Images

Mountains in the background and the V2* dog in front of moongate


Target Images

Georgia o'keeffe style painting of the V2* dog in front of moongate


Ours (Joint)

Ours (Optimization)

DreamBooth

Target Images

The V1* cat looking from inside the V2* wooden pot


Target Images

The V1* cat playing with a V2* wooden pot in a garden


Target Images

A V1* cat engraving on the V2* wooden pot


Target Images

A V1* cat inside the V2* wooden pot


Target Images

The V1* cat is sitting inside a V2* wooden pot and looking up


Target Images

The V1* cat sculpture in the style of a V2* wooden pot


Target Images

The V1* cat is wearing sunglasses and playing with a V2* wooden pot


Target Images

Rose flower in the V2* wooden pot on a table, and the V1* cat is eating the flowers


Ours (Joint)

Ours (Optimization)

DreamBooth

Target Images

Photo of a V1* table and the V2* chair


Target Images

Photo of a V1* table and a V2* chair in the garden


Target Images

V2* chair and V1* table and mountains in the background


Target Images

A V2* chair and a V1* table and a floor lamp on the side


Target Images

A V1* table and a V2* chair near a pool


Target Images

Watercolor painting of a V1* table and a V2* chair


Target Images

A V1* table and a V2* chair, digital illustration


Target Images

Georgia o'keeffe style painting of the V1* table and the V2* chair


Ours (Joint)

Ours (Optimization)

DreamBooth

Target Images

V1* wooden pot with V2* flower in it


Target Images

V2* flower in the V1* wooden pot on a table


Target Images

V2* flower and roses in V1* wooden pot


Target Images

V2* flower engraving on the V1* wooden pot


Target Images

Watercolor painting of V2* flower in V1* wooden pot


Target Images

A digital illustration of V2* flower in V1* wooden pot


Target Images

Mountains in the background and V2* flower in V1* wooden pot


Target Images

Georgia o'keeffe style painting of V2* flower in V1* wooden pot



Results of Ablation and other Experiments


Single-Concept

We show qualitative comparison of our method (2nd column) with Our (compressed model) (3rd column) and Ours (w/ Gen) i.e., generated images as regularization, on 5 prompts per dataset. The compressed model only requires 15MB storage compared to 75MB of original fine-tuned models while having similar sample quality. Our method can also be used with generated images as regularization. This leads to similar sample quality on the target images but some overfitting and worse samples on related concepts as shown in our paper.

Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* dog


Target Images

V* dog swimming in a pool


Target Images

V* dog at a beach with a view of seashore


Target Images

V* dog in times square


Target Images

V* dog wearing sunglasses


Target Images

V* dog in a construction outfit


Target Images

V* dog playing with a ball


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of a moongate


Target Images

Moongate in snowy ice


Target Images

Moongate at a beach with a view of seashore


Target Images

Moongate in the middle of highway road


Target Images

Moongate in times square


Target Images

Moongate on the surface of moon


Target Images

Puppy in front of moongate close up view


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* cat


Target Images

V* cat swimming in a pool


Target Images

V* cat at a beach with a view of seashore


Target Images

V* cat sitting on a window


Target Images

V* cat in times square


Target Images

V* cat wearing sunglasses


Target Images

Construction outfit wearing V* cat


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* table


Target Images

V* table near a pool


Target Images

V* table at a beach with a view of seashore


Target Images

V* table in a garden


Target Images

V* table in grand canyon


Target Images

V* table floating in ocean


Target Images

V* table and an armchair


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* chair


Target Images

V* chair near a pool


Target Images

V* chair at a beach with a view of seashore


Target Images

V* chair in a garden


Target Images

V* chair in grand canyon


Target Images

V* chair in front of a medieval castle


Target Images

V* chair and a coffee table


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* tortoise plushy


Target Images

V* tortoise plushy in grand canyon


Target Images

V* tortoise plushy swimming in pool


Target Images

V* tortoise plushy sitting at the beach with a view of sea


Target Images

V* tortoise plushy in times square


Target Images

V* tortoise plushy in front of a medieval castle


Target Images

V* tortoise plushy wearing sunglasses


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of a V* barn


Target Images

V* barn in snowy ice


Target Images

V* barn in fall season with leaves all around


Target Images

V* barn at a beach with a view of seashore


Target Images

Photo of V* barn with sun rising in the sky


Target Images

Puppy in front of V* barn


Target Images

Cat sitting in front of V* barn


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* flower


Target Images

V* flower growing in desert


Target Images

V* flower at a beach with a view of seashore


Target Images

V* flower on top of a mountain with surise in background


Target Images

A bouquet of V* flower


Target Images

A garden of V* flower


Target Images

V* flower in a violet vase on a table


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* teddybear


Target Images

V* teddybear in grand canyon


Target Images

V* teddybear swimming in pool


Target Images

V* teddybear sitting at the beach with a view of sea


Target Images

V* teddybear in times square


Target Images

V* teddybear wearing sunglasses


Target Images

V* teddybear working on the laptop


Custom Diffusion (Ours)

Ours (5x compressed)

Ours (w/ Gen)

Target Images

Photo of V* wooden pot


Target Images

V* wooden pot in grand canyon


Target Images

V* wooden pot with mountains and sunset in background


Target Images

V* wooden pot floating in a pool


Target Images

A wide shot of V* wooden pot in times square


Target Images

V* wooden pot and chocolate cake on a table


Target Images

Rose flowers in V* wooden pot on a table



Multi-Concept

We compare our Custom-Diffusion (joint training) with the proposed baselines of finetuning all weights in our method and sequential training on multi-concept. Finetuning all weights works worse than our method on multi-concept fine-tuning (as shown quantitatively in the main paper as well). Sequential training results in subpar results on the first concept as most visible on the Moongate + Dog setting.

Ours (Joint)

Ours (seq training)

chair->cat

Ours (w/ fine-tune all)

Target Images

A V2* chair and V1* cat sitting on it


Target Images

A V2* chair and a V1* cat


Target Images

V2* chair with a screaming V1* cat sitting on it


Target Images

V2* chair with the V1* cat sitting on it near a beach


Target Images

V2* chair with the V1* cat sitting near a pool


Target Images

Watercolor painting of V1* cat sitting on V2* chair


Target Images

A digital illustration of V1* cat sitting on V2* chair


Target Images

Mountains in the background and V1* cat is sitting on V2* chair


Ours (Joint)

Ours (seq training)

moongate->dog

Ours (w/ fine-tune all)

Target Images

V2* dog in front of moongate


Target Images

Moongate and V2* dog


Target Images

V2* dog wearing sunglasses in front of moongate


Target Images

V2* dog in front of moongate in snowy ice


Target Images

Watercolor painting of the V2* dog in front of moongate


Target Images

A digital illustration of the V2* dog in front of moongate


Target Images

Mountains in the background and the V2* dog in front of moongate


Target Images

Georgia o'keeffe style painting of the V2* dog in front of moongate


Ours (Joint)

Ours (seq training)

wooden pot->cat

Ours (w/ fine-tune all)

Target Images

The V1* cat looking from inside the V2* wooden pot


Target Images

The V1* cat playing with a V2* wooden pot in a garden


Target Images

A V1* cat engraving on the V2* wooden pot


Target Images

A V1* cat inside the V2* wooden pot


Target Images

The V1* cat is sitting inside a V2* wooden pot and looking up


Target Images

The V1* cat sculpture in the style of a V2* wooden pot


Target Images

The V1* cat is wearing sunglasses and playing with a V2* wooden pot


Target Images

Rose flower in the V2* wooden pot on a table, and the V1* cat is eating the flowers


Ours (Joint)

Ours (seq training)

chair->table

Ours (w/ fine-tune all)

Target Images

Photo of a V1* table and the V2* chair


Target Images

Photo of a V1* table and a V2* chair in the garden


Target Images

V2* chair and V1* table and mountains in the background


Target Images

A V2* chair and a V1* table and a floor lamp on the side


Target Images

A V1* table and a V2* chair near a pool


Target Images

Watercolor painting of a V1* table and a V2* chair


Target Images

A V1* table and a V2* chair, digital illustration


Target Images

Georgia o'keeffe style painting of the V1* table and the V2* chair


Ours (Joint)

Ours (seq training)

wooden pot->flower

Ours (w/ fine-tune all)

Target Images

V1* wooden pot with V2* flower in it


Target Images

V2* flower in the V1* wooden pot on a table


Target Images

V2* flower and roses in V1* wooden pot


Target Images

V2* flower engraving on the V1* wooden pot


Target Images

Watercolor painting of V2* flower in V1* wooden pot


Target Images

A digital illustration of V2* flower in V1* wooden pot


Target Images

Mountains in the background and V2* flower in V1* wooden pot


Target Images

Georgia o'keeffe style painting of V2* flower in V1* wooden pot