Drastic outputs quality drops between Automatic1111 and Hugging face Diffusers

I wanted to move a specific pipeline from Automatic1111 to a pure-code solution using hugging face diffusion pipeline. With the exact same parameter, the quality in the diffusers is worst. it uses instant id, realvisSDXLv40-lightning, guidance scale of 1, same prompt and negative prompt, seed, same instant id parameter, same scheduler (DPM++ 2M Karras).

is there a way to prevent this from happening ?

How to achieve better result with diffusers ?

Thanks in advance

Here are two generation with exactly the same parametters as put above:

Diffusers showcase also a hyper “grainy” outputs it keeps lots of noise in the outputs