I’ve managed to set the additional conditioning on Unet2DModel and trained the model using jupyter notebook.
However, I’m just wondering, is there anyone who managed to train the modified model through CLI?
Running the jupyter notebook through CLI is one of the options, but I’m curious if there could be a more programmatic(?) way.
I have fine tuned using LORA for my PlantVillage Dataset with 15 classes. I did modify my pipeline to take in class labels, and the training loop as well. I was able to train with the options
unet.class_embed_types = None
unet.num_class_embeds = 15
This will internally use a torch.nn.Embedding(num_classes, timestep_projection_dim) and add the embedding to timesteps.
However the results are terrible, all I was able to generate were blurry images, that do not resemble leaves at all.