Is there any way to control the input of a `Longformer` layer?

🤗Transformers
1

Hello,

Is there any way that I can directly control the input to each layer of LongformerForMultipleChoice, similar to GPT2.transformer.h[]?

I tried best_model_longformer.longformer.encoder.layer[layer_index](input_hidden_state_for_layer) but it’s giving this error:

Traceback (most recent call last):
  File "SEED_125_V20_15_LONGFORMER.py", line 426, in <module>
    main_function('/home/ec2-user/G1G2.txt','/home/ec2-user/G1G2_answer_num.txt', num_iter)
  File "SEED_125_V20_15_LONGFORMER.py", line 388, in main_function
    best_model_longformer)
  File "SEED_125_V20_15_LONGFORMER.py", line 205, in fill_MC_loss_accuracy_tensor
    best_model_longformer.longformer.encoder.layer[j](input_hidden_state)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 852, in forward
    output_attentions=output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 796, in forward
    output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 241, in forward
    attention_mask = attention_mask.squeeze(dim=2).squeeze(dim=1)
AttributeError: 'NoneType' object has no attribute 'squeeze'

:S thank you,

2

If you want full control you can just copy paste the code and tweak it the way you want :slight_smile: