Creation of Images from Text-Prompt (Customized Training)

[deicool]

Hello

I want to use ‘Stability Diffusion’ to generate images from text. However I would like to get it trained on my custom images too.

How do I go about doing this?

Please help.

Thank You.

[John6666]

There is also a way to train the entire model for yourself, but personally I think that in most cases, training LoRA is the cheapest way to go.
Well, it is better to learn the basic usage of StableDiffusion first.

LoRA

GitHub - bmaltais/kohya_ss

[deicool]

Sorry, if I just have 100 or so custom images, which way should I be going?

[deicool]

Hello,

I am trying to see if LORA works or not. I have specified the following parameters while training:

Training Images Folder - D:\Ganu\AIImage\huggingface\kohya_ss\data\images
I have put a simple image in the folder (image1.jpeg)

While training using LORA, I get the following error:

“ERROR No data found. Please verify arguments (train_data_dir must be the parent of folders with images) / train_network.py:212
There are no images. Please check the argument specification (train_data_dir must specify the parent folder of the folder with images, not the folder with images)”

Traceback (most recent call last):
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\train_db.py”, line 529, in
train(args)
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\train_db.py”, line 190, in train
train_dataloader = torch.utils.data.DataLoader(
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\torch\utils\data\dataloader.py”, line 349, in init
sampler = RandomSampler(dataset, generator=generator) # type: ignore[arg-type]
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\torch\utils\data\sampler.py”, line 140, in init
raise ValueError(f"num_samples should be a positive integer value, but got num_samples={self.num_samples}“)
ValueError: num_samples should be a positive integer value, but got num_samples=0
Traceback (most recent call last):
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py”, line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py”, line 86, in run_code
exec(code, run_globals)
File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\Scripts\accelerate.EXE_main.py”, line 7, in
sys.exit(main())
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py”, line 47, in main
args.func(args)
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py”, line 1017, in launch_command
simple_launcher(args)
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py”, line 637, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command ‘[‘D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\Scripts\python.exe’, ‘D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/sd-scripts/train_db.py’, ‘–config_file’, ‘D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/outputs/config_dreambooth-20250106-152049.toml’]’ returned non-zero exit status 1.

Any ideas what to do?

[John6666]

In that case, I recommend creating a LoRA. It is said that you can create one with 10 images. Of course, the more images you have, the more reliable it will usually be.
Some people use thousands of images for training models. However, if you have 100 images, I don’t think it’s impossible.

Whether you’re using LoRA or training, the more popular the model you use is and the more suitable it is for the images you’re using, the easier it will be, so I think it’s a good idea to first find a model that is close to your goal. I think that many people use Animagine 3.1, Illustrious, and NoobAI for anime pictures, and SDXL 1.0 and FLUX.1 Dev for live-action pictures. Also, Pony is a type of SDXL that is very popular, but it is special in both good and bad ways, so I think it’s better to avoid it until you get used to it.

[John6666]

The above reply took more than a few days to be reflected, so it’s become a bit of a mystery to everyone…
The script error is difficult to understand, but it seems to be something like this. There is a knack to the folder structure.

’not found data‘error when training

[deicool]

Thank You. I got a demo model trained with your help. Is there a way to use it from the menu:

Kohya_ss setup menu:

1. Install kohya_ss GUI
2. (Optional) Install CuDNN files (to use the latest supported CuDNN version)
3. (DANGER) Install Triton 2.1.0 for Windows… only do it if you know you need it… might break training…
4. (Optional) Install specific version of bitsandbytes
5. (Optional) Manually configure Accelerate
6. (Optional) Launch Kohya_ss GUI in browser
7. Exit Setup

Select an option:

[John6666]

I’ve never actually used kohyass…
If it works, you’re good to go. I think it would be quicker to look for a knowhow article. It’s one of the most popular scripts.

[deicool]

I trained with 1 image and it takes about 3 hours on my GPU. I have about 100 images for training, will it take 100*3 hours?

[John6666]

will it take 100*3 hours?

In theory, yes. But I don’t think people usually spend that much time on it.
I think it’s probably because the settings are insufficient and the GPU isn’t being used much, or the GPU is too weak, or the model you chose is too big and it’s too much for the GPU you have. For example, FLUX is tough if you don’t have two 4090s. There are various tricks you can use… (Quantization…)

[deicool]

Hello,

I trained the model successfully with 1 image and got the following files:
last.safetensors, config_dreambooth-20250108-103158, last_20250108-103158, prompt.

My program with Gradio is as follows:

import torch
from transformers import AutoModel, AutoTokenizer

# Correct paths to your trained model files
config_path = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model/config_dreambooth-20250108-103158"
model_path = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model/last.safetensors"

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(config_path, local_files_only=True)
model = AutoModel.from_pretrained(model_path, config=config_path, local_files_only=True)

# Function to generate an image from a text prompt
def generate_image(prompt):
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
    # Process the outputs to generate the image (example)
    generated_image = outputs[0].cpu().numpy()
    return generated_image

# Example usage
text_prompt = "A beautiful sunset over a tranquil beach"
generated_image = generate_image(text_prompt)

# Save or display the generated image
from PIL import Image
import numpy as np

image = Image.fromarray((generated_image * 255).astype(np.uint8))
image.save("generated_image.jpg")
image.show()

When I try to run the model, I get the following error:

python FirstTry.py

Traceback (most recent call last):
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\transformers\utils\hub.py”, line 403, in cached_file
resolved_file = hf_hub_download(
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\utils_validators.py”, line 106, in _inner_fn
validate_repo_id(arg_value)
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\utils_validators.py”, line 154, in validate_repo_id
raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form ‘repo_name’ or ‘namespace/repo_name’: ‘D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model/config_dreambooth-20250108-103158’. Use repo_type argument if needed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\FirstTry.py”, line 9, in
tokenizer = AutoTokenizer.from_pretrained(config_path, local_files_only=True)
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\transformers\models\auto\tokenization_auto.py”, line 858, in from_pretrained
tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\transformers\models\auto\tokenization_auto.py”, line 690, in get_tokenizer_config
resolved_config_file = cached_file(
File “D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\transformers\utils\hub.py”, line 469, in cached_file
raise EnvironmentError(
OSError: Incorrect path_or_model_id: ‘D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model/config_dreambooth-20250108-103158’. Please provide either the path to a local folder or the repo_id of a model on the Hub.

[John6666]

Hello!

The transformers you are using in your program are libraries for language models and image recognition models, and you use Diffusers and PEFT for image generation models.
Basically, you just need to have LoRA locally. The procedure is to load the base model with from_pretrained, then apply LoRA with load_adapters, and then run inference.
(After applying LoRA, you can save the model as a new model with LoRA applied by running fuse_lora and then save_pretrained, but this is probably not necessary in this case. As long as you have LoRA and the base model, you won’t have any problems with inference.)

In the past, it was necessary to convert LoRA file formats, but now they are basically the same, so this is not necessary.
So, the LoRA file you create can be used as is in A1111 WebUI and ComfyUI.

Load adapters

Load LoRAs for inference

[deicool]

Hello,

Thank you for the reply.

I am trying to run the following program:

from diffusers import AutoPipelineForText2Image
import torch
import os

# Load the base model
pipeline = AutoPipelineForText2Image.from_pretrained("sd-dreambooth-library/herge-style", torch_dtype=torch.float16).to("cuda")

# Specify the path to your LoRA weights file
lora_weights_path = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\last.safetensors"

# Verify the LoRA weights file
if not os.path.exists(lora_weights_path):
    raise FileNotFoundError(f"LoRA weights file not found at {lora_weights_path}")

# Load LoRA weights
try:
    pipeline.load_lora_weights(lora_weights_path)
except ValueError as e:
    raise ValueError("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.") from e

# Generate an image from a text prompt
prompt = "A cute herge_style brown bear eating a slice of pizza, stunning color scheme, masterpiece, illustration"
image = pipeline(prompt).images[0]

# Display the generated image
image.save("generated_image.jpg")
image.show()

I get the following error:

python SeventhTry.py
Loading pipeline components...:  71%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏                                        | 5/7 [00:00<00:00, 10.24it/s]An error occurred while trying to fetch C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\vae: Error no file named diffusion_pytorch_model.safetensors found in directory C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\vae.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
An error occurred while trying to fetch C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\unet: Error no file named diffusion_pytorch_model.safetensors found in directory C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\unet.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00,  4.28it/s]
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\SeventhTry.py", line 17, in <module>
    pipeline.load_lora_weights(lora_weights_path)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\lora_pipeline.py", line 127, in load_lora_weights
    raise ValueError("Invalid LoRA checkpoint.")
ValueError: Invalid LoRA checkpoint.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\SeventhTry.py", line 19, in <module>
    raise ValueError("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.") from e
ValueError: Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.

Any ideas?

[John6666]

The options you give when loading LoRA are a little special. You have to separate the file name from the directory name. Like this.

from diffusers import AutoPipelineForText2Image
import torch
import os
from pathlib import Path

# Load the base model
pipeline = AutoPipelineForText2Image.from_pretrained("sd-dreambooth-library/herge-style", torch_dtype=torch.float16).to("cuda")

# Specify the path to your LoRA weights file
lora_weights_path = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\last.safetensors"

# Verify the LoRA weights file
if not os.path.exists(lora_weights_path):
    raise FileNotFoundError(f"LoRA weights file not found at {lora_weights_path}")

# Load LoRA weights
try:
    pipeline.load_lora_weights(Path(lora_weights_path).parent, weight_name=Path(lora_weights_path).name)
    #pipeline.fuse_lora(lora_scale=1.0) # if LoRA isn't work
    #pipeline.save_pretrained("lora_applied_model") # if you want to save LoRA applied model
    #https://huggingface.co/docs/diffusers/v0.32.1/api/loaders/lora
except ValueError as e:
    raise ValueError("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.") from e

# Generate an image from a text prompt
prompt = "A cute herge_style brown bear eating a slice of pizza, stunning color scheme, masterpiece, illustration"
image = pipeline(prompt).images[0]

# Display the generated image
image.save("generated_image.jpg")
image.show()

[deicool]

Still getting an error:

python SeventhTry.py
# Load the base model
Loading pipeline components...:   0%|                                                                                                                                                       | 0/7 [00:00<?, ?it/s]An error occurred while trying to fetch C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\unet: Error no file named diffusion_pytorch_model.safetensors found in directory C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\unet.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Loading pipeline components...:  71%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏                                        | 5/7 [00:03<00:00,  2.06it/s]An error occurred while trying to fetch C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\vae: Error no file named diffusion_pytorch_model.safetensors found in directory C:\Users\ADMIN\.cache\huggingface\hub\models--sd-dreambooth-library--herge-style\snapshots\33888ec1b1cb44429533e2ce41a9c927c94dff20\vae.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:03<00:00,  2.01it/s]
# Specify the path to your LoRA weights file
# Verify the LoRA weights file
# Load LoRA weights
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\SeventhTry.py", line 22, in <module>
    pipeline.load_lora_weights(Path(lora_weights_path).parent, weight_name=Path(lora_weights_path).name)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\lora_pipeline.py", line 127, in load_lora_weights
    raise ValueError("Invalid LoRA checkpoint.")
ValueError: Invalid LoRA checkpoint.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\SeventhTry.py", line 27, in <module>
    raise ValueError("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.") from e
ValueError: Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.

[John6666]

The error is hidden and I can’t see the cause, so try this.

from diffusers import AutoPipelineForText2Image
import torch
import os
from pathlib import Path

# Load the base model
pipeline = AutoPipelineForText2Image.from_pretrained("sd-dreambooth-library/herge-style", torch_dtype=torch.float16).to("cuda")

# Specify the path to your LoRA weights file
lora_weights_path = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\last.safetensors"

# Verify the LoRA weights file
if not os.path.exists(lora_weights_path):
    raise FileNotFoundError(f"LoRA weights file not found at {lora_weights_path}")

# Load LoRA weights
try:
    pipeline.load_lora_weights(Path(lora_weights_path).parent, weight_name=Path(lora_weights_path).name)
except ValueError as e:
    raise ValueError(f"Invalid LoRA checkpoint. Please check the compatibility and format of the weights file. {e}") from e

[Alanturner2]

Hi there!

To use Stable Diffusion and fine-tune it with your custom images, you can follow these steps:

1. Set Up Stable Diffusion

• Download the pre-trained Stable Diffusion model. You can find the official repository and weights here or use a platform like Hugging Face.
• Ensure you have Python, CUDA-enabled GPUs, and libraries like PyTorch installed.

2. Gather Your Custom Images

• Prepare a dataset of images relevant to your use case.
• Annotate these images if needed (e.g., captions or tags) to help guide the training process.

3. Choose a Fine-Tuning Method

• DreamBooth: Ideal for personalizing the model for specific objects or styles. It fine-tunes the model on a smaller dataset while preserving its overall knowledge.
  • Check out the DreamBooth implementation on Hugging Face for a tutorial.
• LoRA (Low-Rank Adaptation): A lightweight fine-tuning method that requires fewer computational resources.
  • Libraries like PEFT support LoRA integration.
• Textual Inversion: Another lightweight approach that focuses on learning new textual embeddings for your custom concepts.

4. Fine-Tuning Process

• Clone the desired fine-tuning repository, such as:
  • DreamBooth Stable Diffusion
  • Hugging Face Diffusers (supports DreamBooth and LoRA)
• Install the required dependencies.
• Train the model using your images and captions. Adjust parameters like learning rate and training steps for optimal results.

5. Test the Fine-Tuned Model

• After training, generate images using prompts relevant to your custom images.
• Evaluate the quality and adjust training if necessary.

6. Deploy Your Model

• Use tools like Gradio or Streamlit to build an interactive interface for your model.
• Alternatively, deploy it on a server or cloud platform.

Additional Tips:

• Use a GPU with at least 12GB VRAM for efficient training.
• Start with a smaller dataset and fine-tune iteratively to avoid overfitting.
• Explore the Hugging Face Spaces or community notebooks for practical examples.

Hope this help!

[deicool]

Hello

How do I change “Number of Epochs” in kohya_ss from 40 to 1? Can’t find the config file to do the same?

[John6666]

Maybe like these.

Can someone explain "repeats", "epochs" and total steps in Lora training? ·...

How is num_epochs calculated?

LoRA training parameters

[deicool]

Hello

I successfully trained a model (stabilityai/stable-diffusion-2-1-base)

I generated a image using the below program:

from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
import torch
import os
import numpy as np
from PIL import Image

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\"
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
pipeline = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-1-base",
    torch_dtype=torch.float16
).to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path)
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A beautiful Woman"
generated_image = pipeline(prompt=text_prompt).images[0]

# Handle NaN or infinite values and ensure the range is valid 
print("Handle NaN or infinite values and ensure the range is valid ")
generated_image = np.nan_to_num(generated_image, nan=0.0, posinf=255.0, neginf=0.0) 
generated_image = np.clip(generated_image, 0, 255) 
generated_image = generated_image.astype(np.uint8)

# Save or display the generated image
print("Save or display the generated image")

# Convert the NumPy array to a PIL Image and save or display the generated image 
pil_image = Image.fromarray(generated_image) 
pil_image.save("generated_image.jpg") 
pil_image.show()

The image comes out blank with black background. (I tried without Lora weights file, but the image is still blank & black-background)

(Generated image properties:
Shape: (512, 512, 3)
Data type: uint8
Value range: 0 - 0
Handle NaN or infinite values and ensure the range is valid
Save or display the generated image)

Any ideas what to do?