T5-base results are worse than t5-small

Models
1

Hi everyone,

I pretrained T5 small, base and large on the PrivaSeer corpus with a spanned MLM objective. I called the pretrained model PrivaT5. Then finetuned PrivaT5 and T5 small, base and large on some tasks of the PrivacyGLUE benchmark. You can see the results in these plots:

effect_model_size_SC
effect_model_size_SC1280Ă—960 90.5 KB

For all model sizes I used the same hyperparameters except for the batch size I changed it to make the model fit on the TPU. Example :

--model_name_or_path="t5-base"
--hub_save_name_or_path="t5-base"
--model_type="t5-base"
--config_name="t5-base"
--tokenizer_name="t5-base"
--max_seq_length="512"
--per_device_train_batch_size="16"
--per_device_eval_batch_size="16"
--adafactor
--learning_rate="0.001"
--weight_decay="0.0"
--warmup_steps="0"
--overwrite_output_dir
--logging_steps="500"
--save_steps="50"
--eval_steps="50"
--num_train_epochs="100"

Could anyone give me possible reasons why the PrivaT5 base performance unexpectedly drops on the OPP-115 and Policy-Detection tasks compared to PrivaT5 small? (Multilabel text classification & binary text classification respectively).

Thank you!

1 Like