How to export a fine-tuned SDXL model?

[hgfckwla]

I’m trying to use AWS Inferentia 2 (NeuronX) for a more efficient SD-XL image generation. The base model works fine, I can export it to Neuron with this and then use:

optimum-cli export neuron \
  --model stabilityai/stable-diffusion-xl-base-1.0 \
  --task stable-diffusion-xl \
  --batch_size 1  \
  --num_images_per_prompt 4
  sdxl_base_neuron/

But how do I do the same with a finetuned SD XL model, for example with JuggernautXL from CivitAI?

The downloaded file is juggernautXL_v8Rundiffusion.safetensors but when I pass that to the --model parameter it tries to download it from HF. (which it can’t because it’s not a HF-hosted model).

I looked into optimum.exporters.tasks and found that it supports loading models from local dir, but it seems like the models should be in some “expanded” format in the directory. If it’s just the safetensors file it fails with:

OSError: juggernautXL_v8Rundiffusion does not appear 
         to have a file named config.json.

The stable-diffusion-xl-base-1.0 downloaded from HF comes with 4 parts: text_encoder, text_encoder_2, unet, and vae, each having its own config.json.

I’m a bit lost here. How can I convert the juggernaut safetensors file for use with Optimum / Neuron?

[Jingya]

Hi @hgfckwla,

Optimum Neuron is designed to support stable diffusion models via HuggingFace’s Diffusers library. If the checkpoint that you want to convert is not on the HuggingFace hub, then you need to download it to the local, and indicate the path while using optimum CLI(like what you did!). And you would need to make sure that the local models are compatible with Diffusers API.

In the error message you got, it seems that the config file is missing, which is mandatory since we need to deduce some info for the compilation with static shapes.

It seems that Juggernaut X exists on the hub as well, have you tried with: RunDiffusion/Juggernaut-X-v10 · Hugging Face ? Or would it be possible to organize the artifacts that you downloaded from Civitai like the one on the hub to make it compatible with Diffusers thus Optimum Neuron?

[hgfckwla]

Hi @Jingya thanks for your response. Can you point me at a tool that I can use to convert from the standalone safetensors files that can be downloaded from e.g. civit.ai to the format expected by HF Diffusers library?

I understand that the SDXL base from HF contains at least 4 different models: text_encoder, text_encoder_2, unet, and vae, and each of them have their own safetensors file in your repo.

But what are the finetuned safetensors files from civit.ai? Do they bundle all the 4+ models in one file? Or are they just a replacement for one of them, unet perhaps?

Actually this seems to be the sd_xl_base_1.0.safetensors in the same single-file format as the civit.ai models.

If you could help me understand the different formats and how to convert between them that’d be super awesome

Thanks!

[John6666]

This can be done with from_single_file in Diffusers, but there are also tools on HF.
Convert any Stable Diffusion XL checkpoint to Diffusers and open a PR (HF Official)
Download and convert any Stable Diffusion XL safetensors to Diffusers and create your repo (home-made)

[hgfckwla]

Thanks a lot, that’s exactly what I need!