Describe the bug
Hello!
I tried to use train_dreambooth_lora but after finishing training, I can’t see my face on the result image that I trained!
After making an image, that is a very random instance like a Car or train or another human face that is not me.
what is the problem here?!
Reproduction
This is my training parameters:
MODEL_NAME="XpucT/Deliberate"
INSTANCE_DIR="datasets/instance-images/1fdfdf359622486c94e742e3e1a975cb/"
OUTPUT_DIR="output-models/1fdfdf359622486c94e742e3e1a975cb/"
CLASS_DIR="datasets/class-man/"
!accelerate launch train_dreambooth_lora.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--instance_data_dir=$INSTANCE_DIR \
--class_data_dir=$CLASS_DIR \
--output_dir=$OUTPUT_DIR \
--instance_prompt="<hosein>" \
--class_prompt="photo of a man" \
--resolution=512 \
--train_batch_size=1 \
--gradient_accumulation_steps=1 \
--checkpointing_steps=100 \
--with_prior_preservation \
--prior_loss_weight=0.1 \
--learning_rate=1e-7 \
--lr_scheduler="constant" \
--lr_warmup_steps=80 \
--num_class_images=120 \
--max_train_steps=1000 \
--seed="9274" \
--mixed_precision="fp16" \
--train_text_encoder \
--use_8bit_adam \
--enable_xformers_memory_efficient_attention
Now make a pipeline and apply Lora weights:
import torch
import random
from torch import autocast
from diffusers import StableDiffusionPipeline, DDIMScheduler, DPMSolverMultistepScheduler
from IPython.display import display
model_path = "output-models/1fdfdf359622486c94e742e3e1a975cb/"
def get_base_model(model_base):
pipe = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16, safety_checker=None)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")
return pipe
def assign_lora_weight(pipe, lora_weight_id):
pipe.unet.load_attn_procs(lora_weight_id)
return pipe
# get base model for one time
base_pipe = get_base_model("XpucT/Deliberate")
# lora-layer-painter-wise.
hosein_pip = assign_lora_weight(base_pipe, lora_weight_id=model_path)
Now I want make new image from my face:
image = hosein_pip(
"<hosein>",
height=512,
width=512,
negative_prompt="blender, cropped, lowres, poorly drawn face, out of frame, poorly drawn hands, blurry, bad art, blurred, text, watermark, disfigured, blurry background, deformed, closed eyes, animation, CGI ",
num_inference_steps=25,
guidance_scale=7.5,
cross_attention_kwargs={"scale": 2.5},
).images[0]
image.save(f"result/{random.randint(1,1000000)}.jpg")
Logs
No response