Importing community made models and pipeline elements that don't have the diffusers format

🧨 Diffusers
1

Objective

I am currently following the using diffusers section on the documentation page and have come to a point where I can swap out pipeline elements from valid diffusers libraries hosted at hugging face. I can either load the model from the diffusers pipeline and get the components I want, or download and replace the relevant folder manually from the repository. Expanding this, I wanted to try to use community-made stable diffusion models on websites like Civitai, as well as custom community components.

As an example, I would like to use this model checkpoint that is available in a safetensor format:
https://civitai.com/models/7371/rev-animated

With the following vae(s):
https://huggingface.co/WarriorMama777/OrangeMixs/tree/main/VAEs

Problem

I have found that cloning the stable diffusion-v1-5 repository and replacing the .safetensors and .ckpts with the revAnimated '.safetensor` works to run the model.

However, the community vae(s) come either in:

  • .safetensor
  • .vae.pt
  • .ckpt

which is not compatible with the diffusers.models.AutoencoderKL as it expects a config.yaml and diffusion_pytorch_model.bin file. I would like to try to convert these files into a format usable by the diffusers library.

Attempted Solutions

For the conversion I have tried following the instructions on this issue:
https://github.com/huggingface/diffusers/issues/1928

This leads me to the following spaces:
https://huggingface.co/spaces/camenduru/converter
https://huggingface.co/spaces/diffusers/sd-to-diffusers

The first page gets stuck in a preparing space loop:

image
image1473×641 21.4 KB

The other page has a runtime error on load:

Runtime error
Traceback (most recent call last):
  File "app.py", line 3, in <module>
    from convert import convert
  File "/home/user/app/convert.py", line 74, in <module>
    def convert(token: str, model_id: str, filename: str, model_type: str, sample_size: int = 512, scheduler_type: str = "pndm", extract_ema: bool = True, progress=gr.Progress()):
AttributeError: module 'gradio' has no attribute 'Progress'

Queries

  • Is there a way I can do the conversion locally?
  • If yes, does the resulting conversion give me a stable diffusion model or just the vae components?
  • Is there a way to use these components with the diffuser library without the need for conversions?

I am aware that these work out of the box with a custom web UI like Automatic 1111, however, my objective is to better learn the diffusers library.

Thank you for taking the time to go through my topic :smile:

2

hey @Dragonjinx we have conversion scripts for stable diffusion here diffusers/convert_original_stable_diffusion_to_diffusers.py at main · huggingface/diffusers · GitHub

Note that we can’t support arbitrary conversions for arbitrary models and I don’t think we have any recommended vae conversion scripts. You might have to write the code yourself :slight_smile: