I have fine tuned using LORA for my PlantVillage Dataset with 15 classes. I did modify my pipeline to take in class labels, and the training loop as well. I was able to train with the options
unet.class_embed_types = None
unet.num_class_embeds = 15
This will internally use a torch.nn.Embedding(num_classes, timestep_projection_dim) and add the embedding to timesteps.
However the results are terrible, all I was able to generate were blurry images, that do not resemble leaves at all.