How to test pretrained model with already separated-from-whole dataset

Dear @John6666

I tried this but not working with this error

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-38-9b0dbdbf42dd> in <cell line: 0>()
     17 )
     18 
---> 19 trainer_after_train.evaluate()
     20 
     21 

8 frames

/usr/local/lib/python3.11/dist-packages/transformers/data/data_collator.py in torch_default_data_collator(features)
    139         label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
    140         dtype = torch.long if isinstance(label, int) else torch.float
--> 141         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
    142     elif "label_ids" in first and first["label_ids"] is not None:
    143         if isinstance(first["label_ids"], torch.Tensor):

TypeError: must be real number, not PngImageFile

Already trained with the same shape dataset that I have divided for training, validation, and test. Now I need to test the trained model with the test_dataset.

Separation codes:
John6666 previous code

dataset_repo = "seand0101/segformer-b0-finetuned-ade-512-512-manggarai-watergate"
ds = load_dataset(dataset_repo, split="train")
newdsl = []
for i in range(ds.num_rows):
    newdsl.append({"pixel_values": ds[i]["pixel_values"], "label": im.fromarray(np.array(ds[i]["label"].convert("L")) / 255).convert("L")})
newds = Dataset.from_list(newdsl) # normalized dataset

Dividing the dataset

# 70% train, 30% test + validation

train_testvalid = newds.train_test_split(test_size=0.3)

train_testvalid

Dividing it for validation


# 70% train, 30% test + validation
train_testvalid = newds.train_test_split(test_size=0.3)
train_testvalid

Now for the test

# Split the 10% test + valid in half test, half valid

test_valid = train_testvalid['test'].train_test_split(test_size=0.3)

test_valid

Result

# gather everyone if you want to have a single DatasetDict
train_test_valid_dataset = DatasetDict({
    'train': train_testvalid['train'],
    'test': test_valid['test'],
    'valid': test_valid['train']})

Output from result

Codes to validate with the test part of the dataset that came from here and produce such error

hub_model_id_after_train= "seand0101/segformer-b0-finetuned-ade20k-manggarai_rivergate_6"

model_after_train = SegformerForSemanticSegmentation.from_pretrained(
    hub_model_id_after_train,
    num_labels=num_labels,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True, # Will ensure the segmentation specific components are reinitialized.
)

trainer_after_train = Trainer(
    model=model_after_train,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=np.array(test_ds),
    compute_metrics=compute_metrics,
)

trainer_after_train.evaluate()

Did I evaluate it wrongly?
Btw if its not clear, I wanna test the model with part of the dataset so it outputs its own loss plot and accuracy or miou to the test model if that makes anything? or maybe just loss?, is this done correctly?

1 Like

Hello. If you use the same method you used to pass your dataset to Trainer (specifying a custom collate_fn), it should probably work without any problems.
You probably didn’t get an error in metrics and crash during Trainer execution.
That’s because DataCollator (collate_fn) was passing the dataset successfully.

Well, you could also convert the dataset itself to match the default collate_fn in advance, and you should get the same result, so that’s fine too…

Another issue is that whether the evaluation is correct or not ultimately depends on whether the functions related to compute_metrics and Evaluation are suitable for your use case. The computer is just calculating the score according to the instructions.
If the scoring method or target is wrong, the wrong score will be given.
So, once you have got it working, you need to test it and check whether the numbers are reasonable compared to your understanding.

Oh I forgot to erase the np.array part, it’s one of my experiment which of course didn’t work. Previously without np.array part, both train and validation works for training. Now I want to use the testing third of the dataset.

So what happened here

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-29-a577301e3d2f> in <cell line: 0>()
     17 )
     18 
---> 19 trainer_after_train.evaluate()
     20 
     21 

7 frames

/usr/local/lib/python3.11/dist-packages/transformers/data/data_collator.py in torch_default_data_collator(features)
    139         label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
    140         dtype = torch.long if isinstance(label, int) else torch.float
--> 141         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
    142     elif "label_ids" in first and first["label_ids"] is not None:
    143         if isinstance(first["label_ids"], torch.Tensor):

TypeError: must be real number, not PngImageFile
1 Like

When I extract a subset of the dataset using a slice other than the built-in one, it seems to become a Python dict type rather than the Dataset class type.

Is that the reason?

Oh I forgot to erase the np.array part

lol

Edit:
Anyway, the error message is clear, and the process of converting PIL images to PyTorch tensors is missing. It should be the job of collate_fn. It seems that only the data set where the error occurs is not in the format for the default collate_fn.

It’s a dataset data type last time I checked.


It’s confusing lol

Edit:
I don’t remember anything about collate_fn, is it because I use train_test_split(test_size) twice?

1 Like

I just re-read it, and the label_id seems suspicious. Is it mixed datasets up with that?

Edit:

is it because I use train_test_split(test_size) twice?

Possible.

Could you elaborate mixed datasets up, like which one.

Btw I just wanna plot losses of model to the testing dataset, isn’t it just like inference but with array of images at this point? Should I just loop each and plot the loss one by one?

1 Like

It’s just this part. The error is occurring because f[“label”] is not a tensor or a number. Sorry, I was wrong about label_id. I think just the structure of the dataset is still wrong.

--> 141         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)

TypeError: must be real number, not PngImageFile

I found same symptom.

It’s quicker to write collate_fn…

Edit:
I don’t have much experience training models, but I think that the dataset and collate_fn are effectively a set when using Trainer. If either of them changes, it won’t work in the same way.
This is because the format of the data that collate_fn passes to Trainer must be a specific format as Trainer expects. I guess it would be easier if you always made a dataset that works with the default collate_fn, but I don’t know how to do that…

I see thanks then, I will look that up link real quick.

1 Like