Kosmos-2 Fine tuning

[ydshieh]

For anyone other than @Mit1208 (as he knows what’s going on now), the following 2 replies are the 2 changes necessary

Kosmos-2 Fine tuning

See We just need the line inputs[“labels”] = inputs[“input_ids”] as in (note, I haven’t addressed yet the padding and -100 in the labels) from datasets import Dataset inputs = processor(images = train_df['image'].to_list(), text = train_df['text'].to_list(), bboxes = train_df['float_val'].to_list(), padding=True, truncation= True, return_tensors="pt").to(device) inputs["labels"] = inputs["input_ids"] dataset = Dataset.from_dict(inputs) train_test_split = dataset.train_test_split(test_…

Kosmos-2 Fine tuning

I was using attention_mask so it was working with 0 but for input_ids it’s 1. I checked config file, padding token id is 1. new code would look like this: labels = inputs['input_ids'].clone() labels[inputs['input_ids'] == 1] = -100 inputs['labels'] = labels

@cdh Unfortunately, I won’t have the bandwidth to dive into the notebook you provided - especially it contains a lot of custom code and customization.

Regarding the question about labels, see the above 2 links.

For general training with your custom dataset/model, I would recommend:

• try a simple dataset (with just a few text/image paris), train the model (probably with the pretrained one, but you can of course also try from scratch) on that tiny dataset, and see if you can see the loss decreasing, get the model to give the desired generation (on the trained examples)

• Always to to look the examples (before processing, and after being processed by the Kosmos2 processor) , make sure you understand the output of the processor (which are the inputs to the model)

• Once you get familar with the above, think of what would/should be adjusted for your custom dataset and model