Stable diffusion bs>1 uses negative as prompt

[crumb]

Using negative prompts with a batch size > 1 ([prompt]*x , [negative]*x in their respective ways) causes the image to be more towards the negative prompt than the normal prompt.

pipe([prompt]*x, negative_prompt=[negative]*x, width=width, height=height, guidance_scale=guidance_scale, num_inference_steps=steps)

I can link the notebook I’m using if necessary but the code here is so simple that I feel like I don’t need to.
(seed 23335, w/h = 512, prompt = “frog”, negative = “bird”, x = 2, guidance_scale = 7.5, steps = 150, using the lms = LMSDiscreteScheduler( beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear" ))

This might fit better as an issue on the diffusers github but I really don’t know how to use that feature.

that first is using waifu-diffusion v1.3 converting with the conversion script in the repo, but it works the same with the normal stable diffusion v1-4 model too: (linked image so that the “1 embed per new user” thing doesn’t yell at me) https://media.discordapp.net/attachments/1024588665596411944/1028159556561420318/unknown.png?width=1258&height=557

edit: this is fixed by passing both as a single string, not using x or using it as a list, but passing “num_images_per_prompt=x” when calling it, leaving this here in case other people find the same problem (same settings just using num_images_per_prompt=2 instead of making the prompt a list and multiplying by 2: https://media.discordapp.net/attachments/1024588665596411944/1028162978689847396/unknown.png )

[pcuenq]

@crumb

edit: this is fixed by passing both as a single string, not using x or using it as a list, but passing “num_images_per_prompt=x” when calling it

I was going to suggest that, glad you found it.

However, the previous method should also work, I’ll take a look and report back.

Thanks a lot!

[pcuenq]

Hi @crumb,

I was able to reproduce the problem and opened this issue in diffusers to track and fix it.

Thanks a lot for reaching out!

[crumb]

I think this line might have something to do with it?

huggingface/diffusers/blob/f3983d16eed57e46742d217363d8913bef7f748d/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L260

uncond embeddings will already be a list (the negative prompts) so repeating them again won’t help, then in the line after where it’s putting the negatives with the positives, the model is only seeing the negatives (for both the uncond and cond input). Could be wrong but with a quick glance that’s what I think’s happening.

edit: oops you already said something like this in the issue, cheers!