Starting Stable Diffusion In The Middle

Hello. I am playing around with stable diffusion, and I want to be able to start the generation of an image in the middle of the unet where I start with a latent image of noise I have generated

I was looking at the scheduler and there seems to be a diffusion offset parameter. This is close to what I want, however it seems that I would still have to input clip embeddings. Is there anyway to do it just by providing a latent image? Thank you!