Overview
Hi I finetuned from stable diffusion v1.5 with dreambooth method, it worked fine when I trained single subject: car/dog. But the results quality decreased when I trained both categories at the same time with script: train_multi_subject_dreambooth.py
Details
1.Single-subject experiment
--Dog
export MODEL_NAME="/home/mobile360/data/lucien/model/diffusers/stable-diffusion-v1-5"
export INSTANCE_DIR="/home/mobile360/data/lucien/data/task/diffusion-multi/instance/dog3"
export CLASS_DIR="/home/mobile360/data/lucien/data/task/diffusion-multi/class/photo-of-dog"
export OUTPUT_DIR="/home/mobile360/data/lucien/data/task/diffusion-multi/ckpt/output-RPQS-dog-v5"
accelerate launch --mixed_precision="fp16" train_dreambooth.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--instance_data_dir=$INSTANCE_DIR \
--class_data_dir=$CLASS_DIR \
--output_dir=$OUTPUT_DIR \
--with_prior_preservation --prior_loss_weight=1.0 \
--instance_prompt="a photo of RPQS dog" \
--class_prompt="a photo of dog" \
--resolution=512 \
--train_batch_size=1 \
--sample_batch_size=8 \
--gradient_accumulation_steps=2 --gradient_checkpointing \
--use_8bit_adam \
--learning_rate=2e-6 \
--lr_scheduler="constant" \
--lr_warmup_steps=0 \
--num_class_images=400 \
--checkpointing_steps=600 \
--max_train_steps=600
--Car
export MODEL_NAME="/home/mobile360/data/lucien/model/diffusers/stable-diffusion-v1-5"
export INSTANCE_DIR="/home/mobile360/data/lucien/data/task/diffusion-multi/instance/car2"
export CLASS_DIR="/home/mobile360/data/lucien/data/task/diffusion-multi/class/photo-of-car"
export OUTPUT_DIR="/home/mobile360/data/lucien/data/task/diffusion-multi/ckpt/output-TKIB-car-v1"
accelerate launch --mixed_precision="fp16" train_dreambooth.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--instance_data_dir=$INSTANCE_DIR \
--class_data_dir=$CLASS_DIR \
--output_dir=$OUTPUT_DIR \
--with_prior_preservation --prior_loss_weight=1.0 \
--instance_prompt="a photo of TKIB car" \
--class_prompt="a photo of car" \
--resolution=512 \
--train_batch_size=1 \
--sample_batch_size=8 \
--gradient_accumulation_steps=2 --gradient_checkpointing \
--use_8bit_adam \
--learning_rate=2e-6 \
--lr_scheduler="constant" \
--lr_warmup_steps=0 \
--num_class_images=400 \
--checkpointing_steps=600 \
--max_train_steps=600
Results - fine
A photo of RPQS dog
A photo of red TKIB car
2.Multi-subjects experiment
export MODEL_NAME="/home/mobile360/data/lucien/model/diffusers/stable-diffusion-v1-5"
export OUTPUT_DIR="/home/mobile360/data/lucien/data/task/diffusion-multi/ckpt/output-multi-v16"
# Subject 1
export INSTANCE_DIR_1="/home/mobile360/data/lucien/data/task/diffusion-multi/instance/dog4"
export INSTANCE_PROMPT_1="a photo of RPQS dog"
export CLASS_DIR_1="/home/mobile360/data/lucien/data/task/diffusion-multi/class/photo-of-dog/"
export CLASS_PROMPT_1="a photo of dog"
# Subject 2
export INSTANCE_DIR_2="/home/mobile360/data/lucien/data/task/diffusion-multi/instance/car2"
export INSTANCE_PROMPT_2="A photo of TKIB car"
export CLASS_DIR_2="/home/mobile360/data/lucien/data/task/diffusion-multi/class/photo-of-car"
export CLASS_PROMPT_2="A photo of car"
accelerate launch train_multi_subject_dreambooth.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--instance_data_dir="$INSTANCE_DIR_1,$INSTANCE_DIR_2" \
--output_dir=$OUTPUT_DIR \
--instance_prompt="$INSTANCE_PROMPT_1,$INSTANCE_PROMPT_2" \
--with_prior_preservation \
--prior_loss_weight=1.0 \
--class_data_dir="$CLASS_DIR_1,$CLASS_DIR_2" \
--class_prompt="$CLASS_PROMPT_1,$CLASS_PROMPT_2"\
--num_class_images=400 \
--resolution=512 \
--train_batch_size=1 \
--sample_batch_size=8 \
--gradient_checkpointing \
--use_8bit_adam \
--learning_rate=1e-6 \
--lr_scheduler="constant" \
--lr_warmup_steps=0 \
--checkpointing_steps=800 \
--max_train_steps=800
Results - bad
A photo of RPQS dog
A photo of red TKIB car
A TKIB car with a RPQS dog
The results of multi-subjects experiment get worse quality and seem to be fake images.