ControlNet does not work successfully, exist obvious bad style

When I want to use the diffusers/examples/controlnet/ to learn a transform from a road scene of nice weather

to a fog weather

The controller model finally learns to output like this

There exists obvious style and not successful.
But when I use the original code of ControlNet (not in the diffuser), the model can learn the fog effect successfully.

Is there any possible solution I can try?