Model = model.to(args.device) AttributeError: 'function' object has no attribute 'to'

Beginners
1

I am getting this error when trying to do hyperparameter search

Traceback (most recent call last):
  File "/gpfs7kw/linkhome/rech/genlig01/umg16uw/test/expe_5/traitements/gridsearch_flaubert.py", line 581, in <module>
    trainer, outdir = prepare_fine_tuning(PRE_TRAINED_MODEL_NAME, train_dataset, val_dataset, tokenizer, sigle, train_name, datatype, i)
  File "/gpfs7kw/linkhome/rech/genlig01/umg16uw/test/expe_5/traitements/gridsearch_flaubert.py", line 441, in prepare_fine_tuning
    trainer = Trainer(
  File "/linkhome/rech/genlig01/umg16uw/.conda/envs/bert/lib/python3.9/site-packages/transformers/trainer.py", line 366, in __init__
    model = model.to(args.device)
AttributeError: 'function' object has no attribute 'to'
srun: error: r6i3n0: task 0: Exited with exit code 1
srun: Terminating job step 1988679.0

My code :

def model_init():

	set_seed(42)
	mdl = FlaubertModel.from_pretrained(language_model_dir)
	num_labels=3
	# instantiate model

	#model = FlaubertForSequenceClassification.from_pretrained(PRE_TRAINED_MODEL_NAME)
	model = FlauBertForSequenceClassification(config=mdl.config, num_labels=num_labels, freeze_encoder = False)


	# print info about model's parameters
	total_params = sum(p.numel() for p in model.parameters())
	model_parameters = filter(lambda p: p.requires_grad, model.parameters())
	trainable_params = sum([np.prod(p.size()) for p in model_parameters])
	
	return model

		
	
def prepare_fine_tuning(model_name, train_dataset, 
						val_dataset, tokenizer, 
						sigle_model, namefile, data_type, n_tr ):

	"""
	Prepare configurations and base model for fine-tuning
	"""

	# If there's a GPU available...
	if torch.cuda.is_available():    

		# Tell PyTorch to use the GPU.    
		device = torch.device("cuda")

		print('===> There are %d GPU(s) available.' % torch.cuda.device_count())

		print('===> We will use the GPU:', torch.cuda.get_device_name(0))

	# If not...
	else:
		print('===> No GPU available, using the CPU instead.')
		device = torch.device("cpu")
		

	param = str(epoch) + str(bat) + str(l_rate) + str(log_step)
	
	output_dir = outputdir + "/" + "results_hyper" + '/' + sigle_model + '/' + namefile + '/' + data_type + '/' + param
	#print(output_dir)
	logging_dir = outputdir + "/" + "logs" + '/' + sigle_model + '/' + namefile + '/'  + data_type + '/' + param
	#print(logging_dir)


	
	if val_dataset is not None:
			training_args = TrainingArguments(
			output_dir=output_dir,            # output directory
			evaluation_strategy='steps',      # evaluation strategy to adopt during training
			eval_steps=500,                   # number of update steps before evaluation               
			#logging_dir=logging_dir,            # directory for storing logs
			)

			trainer = Trainer(
				model_init=model_init,                         # the instantiated 🤗 Transformers model to be trained
				args=training_args,                  # training arguments, defined above
				train_dataset=train_dataset,         # training dataset
				eval_dataset=val_dataset,            # evaluation dataset
				tokenizer=tokenizer, 			         
				compute_metrics=compute_metrics,
				#callbacks=[EarlyStoppingCallback(3, 0.0)] # early stopping if results dont improve after 3 epochs         
			)
			
			# Default objective is the sum of all metrics
			# when metrics are provided, so we have to maximize it.

			best_trainer = trainer.hyperparameter_search(
			direction="maximize", 
			#backend="ray",  #optuna
			n_trials=10 # number of trials
			)
1 Like
2

Getting a similar error. Did you ever solve this?

3

did you succeed to resolve ? I did not

4

I don’t remember. Sorry. :frowning:

5

This answer might be late, but I had the same problem and it actually took me some time to figure that we put the function name instead of calling the function to create the model. Hence, replace model_init with model_init() in the trainer and it should work. I assume that you followed the tutorial here.
I created a pull request for the notebook as well.

1 Like
6

genius, that solved my problem!, thank you
@tdobrxl

7

Thanks, but are you sure it would be the right thing to do?

model_init should pass in as a callable I believe. See trainer doc - Trainer