To get straight to the point, I think it might be possible to set individual weights for each prompt in the checkpoint model merge.
I am working hard on merging checkpoints in various ways.
I started with weighted merges, of course, and now I am working on hierarchical merges(Merging Block Weights).
I also calculate weights based on the statistics of some of my favorite merge models that I have created in the past.
Along the way, I also studied the image generation process of StableDiffusion.
If I am correct, the resolution of the images handled at each of the three layers varies greatly, with the deeper layers processing coarser images.
Take for example a person with a MEDIUM FULL SHOT angle of view,
In the redeep layer, it looks like a very rough mosaic has been created, so the face is not recognizable at all, but it seems to have an affect on posing and other general aspects of the image.
In the middle layer, the placement of facial parts is affected.
The shallow layer affects details such as the expression of the eyes, the way the eyelashes are drawn, and the texture of the material.
In other words, if you focus on the medium full shot and merge it, you will end up with a completely different face because of the different weights in the do-up and pull angles.
Here’s a thought…
If the flow of image generation is that CLIP first converts the prompts and then creates the corresponding images at each level, wouldn’t it be possible to merge the models with different weights for each individual at the first CLIP converted stage?
First select the two models you want to merge and determine the base weights.
Then you can set the weights of the phrases (lineages) listed below individually and merge them.
Of course, there will be a huge number of prompts, but once you have a certain amount of systematization (enough to satisfy most people) at the initial stage
The number of items that need to be adjusted during the merging process can be reduced considerably.
I think we can work out how to systematize them by gathering the opinions of everyone on the forum.
For each item, you can enter the weight using a slide bar or numerical input, and if there are other words or phrases that you want to set a special weight for, you can use a format such as “word: weight” to enter the weight.
I tried to ask chatGPT to make it, but I have to give up because I don’t know enough about programming.
Is there anyone out there that can make this happen?
Or if it already exists, please let me know.
Translated with DeepL.com (free version)