Using the code below, I train stable diffusion model on 20 images. at inference, I tried different interference steps and I realize that model is overfitted to particularly one photo in the training set and sometime I have multiple heads in one photos. can you give me some orientation of what I can do differently?
Second question: I realize that many are using ShivamShrirao GitHub fork of diffuser library. Why many choose to use it instead of main Huggingface library?
`!accelerate launch train_dreambooth.py \
`MODEL_NAME="stabilityai/stable-diffusion-2-1"`
`INSTANCE_DIR="/mnt/c/Users/noobw/PycharmProjects/pythonProject/Anas"`
`OUTPUT_DIR="/mnt/c/Users/noobw/PycharmProjects/pythonProject/dreambooth_anas"`
`!accelerate launch train_dreambooth.py \`
`--pretrained_model_name_or_path=$MODEL_NAME \`
`--instance_data_dir=$INSTANCE_DIR \`
`--output_dir=$OUTPUT_DIR \`
`--instance_prompt="a photo of arzk man" \`
`--resolution=512 \`
`--train_batch_size=1 \`
`--gradient_accumulation_steps=2 \`
`--learning_rate=5e-6 \`
`--lr_scheduler="constant" \`
`--lr_warmup_steps=0 \`
`--max_train_steps=1200 \`
`--use_8bit_adam \`
`--gradient_checkpointing \`
`--enable_xformers_memory_efficient_attention \`
`--train_text_encoder \`
`--set_grads_to_none \`
`--validation_prompt="a photo of arzk man"\`
`--num_validation_images=4\`
`--validation_steps=100`
inference using :
unet = UNet2DConditionModel.from_pretrained("/mnt/c/Users/noobw/PycharmProjects/pythonProject/dreambooth_anas/checkpoint-{}/unet".format(g))
text_encoder = CLIPTextModel.from_pretrained("/mnt/c/Users/noobw/PycharmProjects/pythonProject/dreambooth_anas/checkpoint-{}/text_encoder".format(g))
pipeline = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1", unet=unet, text_encoder=text_encoder, dtype=torch.float16, #text_encoder=text_encoder,
).to("cuda")
prompt= 'a photo of rzk man, close-up'
image = pipeline(prompt= prompt, num_inference_steps=30, guidance_scale=7.5).images[0]
image.show()
