Error while training LORA in KOHYA_SS (stabilityai/stable-diffusion-xl-base-1.0)

Beginners
1

Hello

I have provided 1 image for training. It’s a simple JPG.

Sample1
Sample1250×200 11.3 KB

I am getting the error below:

Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\sdxl_train_network.py", line 185, in <module>
    trainer.train(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\train_network.py", line 272, in train
    train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 2325, in cache_latents
    dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 1146, in cache_latents
    cache_batch_latents(vae, cache_to_disk, batch, subset.flip_aug, subset.alpha_mask, subset.random_crop)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\sd-scripts\library\train_util.py", line 2775, in cache_batch_latents
    raise RuntimeError(f"NaN detected in latents: {info.absolute_path}")
RuntimeError: NaN detected in latents: D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\trained-model\img\40_A Event of a Pharma Client A Event of a Pharma Client\Sample1.jpg
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module>
    sys.exit(main())
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
    args.func(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\venv\\Scripts\\python.exe', 'D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', 'D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model/config_lora-20250130-144011.toml']' returned non-zero exit status 1.
14:43:08-514022 INFO     Training has ended.

The config file is as below:

bucket_no_upscale = true
bucket_reso_steps = 64
cache_latents = true
caption_extension = ".txt"
clip_skip = 1
dynamo_backend = "no"
enable_bucket = true
epoch = 1
gradient_accumulation_steps = 1
huber_c = 0.1
huber_schedule = "snr"
learning_rate = 0.0001
logging_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\log"
loss_type = "l2"
lr_scheduler = "cosine"
lr_scheduler_args = []
lr_scheduler_num_cycles = 1
lr_scheduler_power = 1
lr_warmup_steps = 160
max_bucket_reso = 2048
max_data_loader_n_workers = 0
max_grad_norm = 1
max_timestep = 1000
max_token_length = 75
max_train_steps = 1600
min_bucket_reso = 256
mixed_precision = "fp16"
multires_noise_discount = 0.3
network_alpha = 1
network_args = []
network_dim = 8
network_module = "networks.lora"
noise_offset_type = "Original"
optimizer_args = []
optimizer_type = "AdamW8bit"
output_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model"
output_name = "last"
pretrained_model_name_or_path = "stabilityai/stable-diffusion-xl-base-1.0"
prior_loss_weight = 1
resolution = "512,512"
sample_prompts = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\model\\prompt.txt"
sample_sampler = "euler_a"
save_every_n_epochs = 1
save_model_as = "safetensors"
save_precision = "fp16"
text_encoder_lr = 0.0001
train_batch_size = 1
train_data_dir = "D:/Ganu/AIImage/huggingface/kohya_ss/kohya_ss/trained-model\\img"
unet_lr = 0.0001
xformers = true
1 Like
2

Maybe this is it. There is a famous bug in the VAE of stabilityai/stable-diffusion-xl-base-1.0, and it seems that the VAE does not work properly with fp16 precision…

1 Like
3

Hello

I successfully trained the model. Your help resolved it.

Now when I run the model to generate a new image, I get the following error:

python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  7.55it/s]
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
Load the LoRA weights
Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 34, in <module>
    raise e
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 31, in <module>
    pipeline.load_lora_weights(lora_weights_path)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\lora_pipeline.py", line 129, in load_lora_weights
    self.load_lora_into_unet(
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\lora_pipeline.py", line 305, in load_lora_into_unet
    unet.load_lora_adapter(
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\loaders\peft.py", line 301, in load_lora_adapter
    inject_adapter_in_model(lora_config, self, adapter_name=adapter_name, **peft_kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\mapping.py", line 260, in inject_adapter_in_model
    peft_model = tuner_cls(model, peft_config, adapter_name=adapter_name, low_cpu_mem_usage=low_cpu_mem_usage)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\tuners\lora\model.py", line 141, in __init__
    super().__init__(model, config, adapter_name, low_cpu_mem_usage=low_cpu_mem_usage)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\tuners\tuners_utils.py", line 184, in __init__
    self.inject_adapter(self.model, adapter_name, low_cpu_mem_usage=low_cpu_mem_usage)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\peft\tuners\tuners_utils.py", line 520, in inject_adapter
    raise ValueError(error_msg)
ValueError: Target modules {'8.1.transformer_blocks.5.attn2.to_v', '8.1.transformer_blocks.8.attn1.to_out.0', 'mid_block.1.transformer_blocks.8.attn1.to_k', '1.1.transformer_blocks.0.attn2.to_q', '8.1.transformer_blocks.9.attn1.to_q', 'mid_block.1.proj_out', '7.1.transformer_blocks.5.attn2.to_out.0', 'mid_block.1

My program is as follows:

from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
#lora_weights_path = os.path.join(model_dir, "last.safetensors")
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
hf_token = "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"
pipeline = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float32,
).to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path)
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]

# Convert the generated image to a NumPy array
generated_image_np = np.array(generated_image)

# Check the generated image properties
print("Generated image properties:")
print("Shape:", generated_image_np.shape)
print("Data type:", generated_image_np.dtype)
print("Value range:", generated_image_np.min(), "-", generated_image_np.max())

# Handle NaN or infinite values and ensure the range is valid
print("Handle NaN or infinite values and ensure the range is valid")
generated_image_np = np.nan_to_num(generated_image_np, nan=0.0, posinf=255.0, neginf=0.0)
generated_image_np = np.clip(generated_image_np, 0, 255)
generated_image_np = generated_image_np.astype(np.uint8)

# Save or display the generated image
print("Save or display the generated image")

# Convert the NumPy array back to a PIL Image and save or display the generated image
pil_image = Image.fromarray(generated_image_np)
pil_image.save("generated_image.jpg")
pil_image.show()
1 Like
4

Hello. It seems to be a compatibility issue with the Diffusers.:sweat_smile: It is said to work after converting.

5

Hello

I ran the below program and getting a blank black image.

The code

from diffusers import AutoPipelineForText2Image
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
#lora_weights_path = os.path.join(model_dir, "last.safetensors")
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")


# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]

# Convert the image data to a NumPy array
image_data = np.array(generated_image)

# Handle NaN or infinite values
image_data = np.nan_to_num(image_data, nan=0.0, posinf=255.0, neginf=0.0)

# Ensure the values are within the range [0, 255]
image_data = np.clip(image_data, 0, 255)

# Convert to uint8
image_data = image_data.astype(np.uint8)

# Create a PIL image from the NumPy array
final_image = Image.fromarray(image_data)

# Save and display the final image
final_image.save("generated_image.png")
final_image.show()

The output

python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00,  3.23it/s]
Load the LoRA weights
Generate an image from a text prompt
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [45:56<00:00, 55.14s/it]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\image_processor.py:147: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

The image is below

kohya_ss_output_stable-diffusion-xl-base-1.0
kohya_ss_output_stable-diffusion-xl-base-1.01024×1024 207 Bytes

1 Like
6

I generated this image by like this

from diffusers import AutoPipelineForText2Image
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
#lora_weights_path = os.path.join(model_dir, "last.safetensors")
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]
generated_image.save("generated_image.png")

generated_image
generated_image256×256 15.1 KB

7

Strange, I used your code, but still not works with me (I updated all libraries too)

python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00,  2.57it/s]
Load the LoRA weights
Generate an image from a text prompt
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [41:52<00:00, 50.24s/it]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\image_processor.py:147: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")
1 Like
8

There is a problem with float16 when used with older GeForce cards, and the image may turn black, I wonder if that’s it.

#, torch_dtype=torch.float16
, torch_dtype=torch.float32
9

This might not concern you, but getting the following error:

python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:43<00:00,  6.22s/it]
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 26, in <module>
    pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16").to("cuda")
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 461, in to
    module.to(device, dtype)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\models\modeling_utils.py", line 1077, in to
    return super().to(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1340, in to
    return self._apply(convert)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply
    module._apply(fn)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply
    module._apply(fn)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 900, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 927, in _apply
    param_applied = fn(param)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1326, in convert
    return t.to(
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 4.00 GiB of which 0 bytes is free. Of the allocated memory 10.67 GiB is allocated by PyTorch, and 229.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
1 Like
10

This way, the shortage in VRAM is made up for with RAM. It’s slow, but…

#pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16", device_map="auto")
11 
python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 26, in <module>
    pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16", device_map="auto")
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\auto_pipeline.py", line 428, in from_pretrained
    return text_2_image_cls.from_pretrained(pretrained_model_or_path, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\utils\_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 710, in from_pretrained
    raise NotImplementedError(
NotImplementedError: auto not supported. Supported strategies are: balanced

And the “balanced” strategy is giving me the following error

python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
Loading pipeline components...:  71%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏                                        | 5/7 [00:37<00:12,  6.27s/it]Some parameters are on the meta device because they were offloaded to the cpu.
Some parameters are on the meta device because they were offloaded to the cpu.
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:38<00:00,  5.54s/it]
Load the LoRA weights
You shouldn't move a model that is dispatched using accelerate hooks.
You shouldn't move a model that is dispatched using accelerate hooks.
Generate an image from a text prompt
  0%|                                                                                                                                                                                      | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "D:\Ganu\AIImage\huggingface\kohya_ss\kohya_ss\user\John-Training-30thJan2025.py", line 42, in <module>
    generated_image = pipeline(prompt=text_prompt).images[0]
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py", line 1208, in __call__
    noise_pred = self.unet(
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\models\unets\unet_2d_condition.py", line 1143, in forward
    emb = self.time_embedding(t_emb, timestep_cond)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\models\embeddings.py", line 1304, in forward
    sample = self.linear_1(sample)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\torch\nn\modules\linear.py", line 125, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
1 Like
12

If the problem can’t be solved by upgrading to a newer version of accelerate, this method is going to be difficult…

pip install -U accelerate diffusers
13

I remembered something important. There is a bug in the official SDXL 1.0 VAE with 16-bit precision, so this may be improved.

from diffusers import AutoPipelineForText2Image, AutoencoderKL
import torch
import os
import numpy as np
from PIL import Image

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, vae=vae, torch_dtype=torch.float16, variant="fp16").to("cuda")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]
generated_image.save("generated_image.png")
14

Getting same error:

python John-Training-30thJan2025.py
Define the path to the directory containing your model and LoRA weights
Load the base model using StableDiffusionPipeline
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 631/631 [00:00<00:00, 630kB/s]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\huggingface_hub\file_download.py:140: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\ADMIN\.cache\huggingface\hub\models--madebyollin--sdxl-vae-fp16-fix. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
diffusion_pytorch_model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 335M/335M [00:45<00:00, 7.41MB/s]
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:02<00:00,  2.72it/s]
Load the LoRA weights
Generate an image from a text prompt
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [43:16<00:00, 51.93s/it]
D:\Ganu\AIImage\huggingface\kohya_ss\Python310\lib\site-packages\diffusers\image_processor.py:147: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")
1 Like
15

If it comes to that, it has to be this…

17

John6666,

The following code gives the same error:

from diffusers import AutoPipelineForText2Image, AutoencoderKL
import torch
import os
import numpy as np
from PIL import Image

print("vae")

# Clear GPU memory before starting 
torch.cuda.empty_cache() 

# Set seed for reproducibility 
#torch.manual_seed(6666666) 
#np.random.seed(6666666)

# Define the path to the directory containing your model and LoRA weights
print("Define the path to the directory containing your model and LoRA weights")
model_dir = "D:\\Ganu\\AIImage\\huggingface\\kohya_ss\\kohya_ss\\trained-model\\model\\" 
lora_weights_path = os.path.join(model_dir, "last.safetensors")

# Load the base model using StableDiffusionPipeline
print("Load the base model using StableDiffusionPipeline")
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
adapter_id = "wangfuyun/PCM_SDXL_LoRAs"

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipeline = AutoPipelineForText2Image.from_pretrained(model_id, vae=vae, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipeline.enable_sequential_cpu_offload()
pipeline.enable_attention_slicing("max")

# Load the LoRA weights
print("Load the LoRA weights")
try:
    pipeline.load_lora_weights(lora_weights_path, weight_name="last.safetensors")
except ValueError as e:
    print("Invalid LoRA checkpoint. Please check the compatibility and format of the weights file.")
    raise e

# Generate an image from a text prompt
print("Generate an image from a text prompt")
text_prompt = "A Event of a Pharma Client"
generated_image = pipeline(prompt=text_prompt).images[0]
generated_image.save("generated_image.png")
1 Like
18

I am not training, but generating images from a trained model (The training is successfully completed)

1 Like
19

In that case, it seems likely that the problem is not due to a bug in VAE, but rather a limitation of GeForce 10x0 and/or torch.float16. I think it would work if you let the CPU do the inference, but the speed is too slow…

pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16").to("cpu")

or

pipeline = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=torch.float32, variant="fp16")
pipeline.enable_model_cpu_offload(device="cuda")

?

If you quantize it, you can save VRAM, but actually, there is a bug where LoRA will not work if you do that…

21

Works. Thank You.

1 Like
22

Hello John6666,

A different question:

The output is coming great for the prompt “A Beautiful Woman” (I have trained for 10 images).

ABeautifulWoman-11thFeb2025-2
ABeautifulWoman-11thFeb2025-2387×387 122 KB

However the output for the prompt “A Event of a Pharma Client” is not that all good ( I have trained for 10 images).

PharmaEvent-11thFeb2025
PharmaEvent-11thFeb2025614×614 28.5 KB

Why would that be?

Thank You

1 Like