Customizing model architecture from predefined models

🤗Transformers
1

Suppose that we want to modify some architecture in the transformers library where should we star?
Are there any tutorials or notebook examples?

For instance I wanted to modify an encoder-decoder model and use only the decoder part.
What should be taken into account in terms of inputs-outputs data pre-post processing.

I tried sub classing the model such as bart or t5 and then use only the encoder part in but that didn’t work.

class T5DecoderOnly(T5ForConditionalGeneration):
    def __init__(self, config):
        super().__init__(config)
        self.model_dim = config.d_model
        self.base_model = self.encoder

model = T5DecoderOnly(config # from pretrained T5)

When printing print(model) it show both encoder and decoder in the architecture.

Also, tried the following:

model = T5ForConditionalGeneration(config)
model = torch.nn.Sequential(*[model.decoder, model.lm_head]

But, again this times throws an error because this time the forward of the decoder does not recognise the arguments input_ids and labels, so calling model(input_ids=input_ids, labels=label_ids) will not work.

I tried to concat inputs and labels and see what happens model(torch.cat([input_ids, label_ids], dim=1))

This time I get a
TypeError: linear(): argument 'input' (position 1) must be Tensor, not BaseModelOutputWithPastAndCrossAttentions

Verbose error output:

TypeError                                 Traceback (most recent call last)
Cell In[148], line 77
     73 label_ids = tokenizer.batch_encode_plus(labels, padding=True, return_tensors="pt", truncation=True)["input_ids"].to(device)
     75 # Forward pass
     76 # outputs = model(input_ids=input_ids, labels=label_ids)
---> 77 outputs = model(torch.cat([input_ids, label_ids], dim=1)) # decoder maybe doesn't take labels arguments and need to be concat with inputs
     78 loss = outputs.loss
     80 # Backward pass

File /idiap/temp/imitro/miniconda3/envs/length-gen/lib/python3.12/site-packages/torch/nn/modules/module.py:1511, in Module._wrapped_call_impl(self, *args, **kwargs)
   1509     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1510 else:
-> 1511     return self._call_impl(*args, **kwargs)

File /idiap/temp/imitro/miniconda3/envs/length-gen/lib/python3.12/site-packages/torch/nn/modules/module.py:1520, in Module._call_impl(self, *args, **kwargs)
   1515 # If we don't have any hooks, we want to skip the rest of the logic in
   1516 # this function, and just call forward.
   1517 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1518         or _global_backward_pre_hooks or _global_backward_hooks
   1519         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1520     return forward_call(*args, **kwargs)
   1522 try:
   1523     result = None

File /idiap/temp/imitro/miniconda3/envs/length-gen/lib/python3.12/site-packages/torch/nn/modules/container.py:217, in Sequential.forward(self, input)
    215 def forward(self, input):
    216     for module in self:
--> 217         input = module(input)
    218     return input

File /idiap/temp/imitro/miniconda3/envs/length-gen/lib/python3.12/site-packages/torch/nn/modules/module.py:1511, in Module._wrapped_call_impl(self, *args, **kwargs)
   1509     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1510 else:
-> 1511     return self._call_impl(*args, **kwargs)

File /idiap/temp/imitro/miniconda3/envs/length-gen/lib/python3.12/site-packages/torch/nn/modules/module.py:1520, in Module._call_impl(self, *args, **kwargs)
   1515 # If we don't have any hooks, we want to skip the rest of the logic in
   1516 # this function, and just call forward.
   1517 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1518         or _global_backward_pre_hooks or _global_backward_hooks
   1519         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1520     return forward_call(*args, **kwargs)
   1522 try:
   1523     result = None

File /idiap/temp/imitro/miniconda3/envs/length-gen/lib/python3.12/site-packages/torch/nn/modules/linear.py:116, in Linear.forward(self, input)
    115 def forward(self, input: Tensor) -> Tensor:
--> 116     return F.linear(input, self.weight, self.bias)

Any ideas how to resolve this?