DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation Using Limited Data

Abstract

Denoising diffusion probabilistic models (DDPMs) have been proven capable of synthesizing high-quality images with remarkable diversity when trained on large amounts of data. Unfortunately, they are still vulnerable to overfitting when fine-tuned on limited data. Existing works have explored subject-driven generation with text-to-image (T2I) models using a few samples. However, there is still a lack of effective and stable data-efficient methods to synthesize images in specific domains (e.g. styles or properties), which remains challenging due to ambiguities inherent in natural language and out-of-distribution effects. This paper introduces a few-shot fine-tuning approach named DomainStudio as a domain-driven image generation paradigm, which is designed to retain the subjects from prior knowledge provided by pre-trained models and adapt them to the domain extracted from training data, pursuing high quality and great diversity. We propose to keep the image-level relative distances between adapted samples and enhance the learning of high-frequency details from both pre-trained models and training samples. DomainStudio is compatible with both unconditional and T2I DDPMs. The proposed method achieves better results than current state-of-the-art GAN-based approaches in unconditional few-shot image generation. It also outperforms existing few-shot fine-tuning methods for modern large-scale T2I diffusion models like Textual Inversion and DreamBooth on synthesizing samples in specific domains characterized by few-shot training data.

Appendices

Appendix A Unconditional Source Models

We train DDPMs on FFHQ $256^2$ (Karras et al., 2020) and LSUN Church $256^2$ (Yu et al., 2015) from scratch for 300K iterations and 250K iterations as source models for DDPM adaptation, which cost 5 days and 22 hours, 4 days and 22 hours on $\times 8$ NVIDIA RTX A6000 GPUs, respectively. We randomly sample 1000 images with these two models to evaluate their generation diversity using the average pairwise LPIPS (Zhang et al., 2018) metric, as shown in Table 9.

Table 9 Average pairwise LPIPS ($\uparrow$) results of 1000 samples produced by StyleGAN2 and DDPMs trained on FFHQ $256^2$ and LSUN Church $256^2$

For comparison, we also evaluate the generation diversity of the source StyleGAN2 (Karras et al., 2020) models used by GAN-based baselines (Wang et al., 2018; Karras et al., 2020; Mo et al., 2020; Wang et al., 2020; Li et al., 2020; Ojha et al., 2021; Zhao et al., 2022a). DDPMs trained on FFHQ $256^2$ and LSUN Church $256^2$ achieve generation diversity similar to the widely-used StyleGAN2 models. Besides, we sample 5000 images to evaluate the generation quality of the source models using FID (Heusel et al., 2017). As shown in Table 10, DDPM-based source models achieve FID results similar to StyleGAN2 on the source datasets FFHQ $256^2$ and LSUN Church $256^2$.

Table 10 FID ($\downarrow$) results of StyleGAN2 and DDPMs trained on FFHQ $256^2$ and LSUN Church $256^2$

Appendix B Additional Ablation Analysis

We provide detailed ablation analysis of the weight coefficients of $\mathcal {L}_{img}$, $\mathcal {L}_{hf}$, and $\mathcal {L}_{hfmse}$ using 10-shot FFHQ $\rightarrow$ Babies (unconditional) as an example. Intra-LPIPS and FID are employed for quantitative evaluation.

We first ablate $\lambda _2$, the weight coefficient of $\mathcal {L}_{img}$. We adapt the source model to 10-shot Babies without $\mathcal {L}_{hf}$ and $\mathcal {L}_{hfmse}$. The quantitative results are listed in Table 11. Corresponding generated samples are shown in Fig. 16. When $\lambda _2$ is set as 0.0, the directly fine-tuned model produces coarse results lacking high-frequency details and diversity. With an appropriate choice of $\lambda _2$, the adapted model achieves greater generation diversity and better learning of target distributions under the guidance of $\mathcal {L}_{img}$. Too large values of $\lambda _2$ make $\mathcal {L}_{img}$ overwhelm $\mathcal {L}_{simple}$ and prevent the adapted model from learning target distributions, leading to degraded generation quality and diversity. The adapted model with $\lambda _2$ value of 2.5 gets unnatural generated samples even if it achieves the best FID result. We recommend $\lambda _2$ ranging from 0.1 to 1.0 for the unconditional adaptation setups used in our paper based on a comprehensive consideration of the qualitative and quantitative evaluation.

Table 11 Intra-LPIPS ($\uparrow$) and FID ($\downarrow$) results of adapted models trained on 10-shot FFHQ $\rightarrow$ Babies with different $\lambda _2$, the weight coefficient of $\mathcal {L}_{img}$

Fig. 16

Visualized ablations of $\lambda _2$, the weight coefficient of $\mathcal {L}_{img}$ on 10-shot FFHQ $\rightarrow$ Babies

Next, we ablate $\lambda _3$, the weight coefficient of $\mathcal {L}_{hf}$ with $\lambda _2$ set as 0.5. The quantitative results are listed in Table 12. Corresponding generated samples are shown in Fig. 17. $\mathcal {L}_{hf}$ guides adapted models to keep diverse high-frequency details learned from source samples for more realistic results. $\mathcal {L}_{hf}$ helps the adapted model enhance details like clothes and hairstyles and achieves better FID and Intra-LPIPS, indicating improved quality and diversity. Too large values of $\lambda _3$ make the adapted model pay too much attention to high-frequency components and fail to produce realistic results following the target distributions. We recommend $\lambda _3$ ranging from 0.1 to 1.0 for the unconditional adaptation setups used in our paper.

Table 12 Intra-LPIPS ($\uparrow$) and FID ($\downarrow$) results of adapted models trained on 10-shot FFHQ $\rightarrow$ Babies with different $\lambda _3$, the weight coefficient of $\mathcal {L}_{hf}$

Fig. 17

Visualized ablations of $\lambda _3$, the weight coefficient of $\mathcal {L}_{hf}$ on 10-shot FFHQ $\rightarrow$ Babies

Table 13 Intra-LPIPS ($\uparrow$) and FID ($\downarrow$) results of adapted models trained on 10-shot FFHQ $\rightarrow$ Babies with different $\lambda _4$, the weight coefficient of $\mathcal {L}_{hfmse}$

Fig. 18

Visualized ablations of $\lambda _4$, the weight coefficient of $\mathcal {L}_{hfmse}$ on 10-shot FFHQ $\rightarrow$ Babies

Fig. 19

Qualitative comparison between LoRA, DreamBooth, and DomainStudio

Finally, we ablate $\lambda _4$, the weight coefficient of $\mathcal {L}_{hfmse}$, with $\lambda _2$ and $\lambda _3$ set as 0.5. The quantitative results are listed in Table 13. Corresponding generated samples are shown in Fig. 18. $\mathcal {L}_{hfmse}$ guides the adapted model to learn more high-frequency details from limited training data. Appropriate choice of $\lambda _4$ helps the adapted model generate diverse results containing rich details. Besides, the full DomainStudio approach achieves state-of-the-art results of FID and Intra-LPIPS on 10-shot FFHQ $\rightarrow$ Babies (see Table 2 and 1). Similar to $\lambda _2$ and $\lambda _3$, too large values of $\lambda _4$ lead to unreasonable results deviating from the target distributions. We recommend $\lambda _4$ ranging from 0.01 to 0.08 for the unconditional adaptation setups in this paper. Results in Fig. 16, 17, and 18 are synthesized from fixed noise inputs.

Fig. 20

1-shot T2I generation samples of DomainStudio given different text prompts

Appendix C Additional Visualized Samples

As illustrated in Sec. 4, DomainStudio is capable of adapting the subjects prompted in text prompts to the style of few-shot training samples. However, baselines like DreamBooth (Ruiz et al., 2023) and Textual Inversion (Gal et al., 2023) fail to produce reasonable samples. We employ LoRA (Hu et al., 2021) as another baseline for T2I generation and provide qualitative results in Fig. 19. LoRA also suffers from overfitting or underfitting in domain-driven generation tasks like DreamBooth. Fig. 20 shows additional results of DomainStudio on T2I generation using a single image as training data.