ControlNet does not work successfully, exist obvious bad style

🧨 Diffusers
1

When I want to use the diffusers/examples/controlnet/train_controlnet.py to learn a transform from a road scene of nice weather

image
image1506×547 60.1 KB

2

to a fog weather
image

3

The controller model finally learns to output like this

image
image1078×392 30.4 KB

There exists obvious style and not successful.
But when I use the original code of ControlNet (not in the diffuser), the model can learn the fog effect successfully.

Is there any possible solution I can try?