Is there a workflow that helps speeding up the loading of SDXL LoRAs using Diffusers? I am using Runpod Serverless and about half of the runtime is spent loading in different LoRAs. It seems like there should be a quicker way to load the LoRAs as they have already been downloaded into memory and the loading of the pipeline only takes around 1 second vs 5 seconds per LoRA.
Any help would be greatly appreciated!