Approach for changing hairstyles in images realistically

What should be an approach to building a model where I have a input image and a reference hairstyle image and I want to transfer the hairstyle from the reference image to input image realistically.

The closest I could find was masking the hair region and use prompts to generate new hairstyle which is fine but the problem is for the same prompt it generates new images every time. What should I do ensure everytime I am changing the hairstyle from reference image, only that hairstyle is inpainted on the target image?