I have a finetuned model based on T5-Base but in order to use that in production I need to compile it using torchscript. But it seems that T5-Base model is not torchscriptable as it can be seen from the following error that I get:
import torch from transformers import ( T5ForConditionalGeneration, ) model = T5ForConditionalGeneration.from_pretrained('t5-base', torchscript=True) torch.jit.script(model)
The error message:
torch.jit.frontend.UnsupportedNodeError: function definitions aren't supported: File "/home/pya/anaconda3/envs/ecg_env/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 984 use_cache = False def create_custom_forward(module): ~~~ <--- HERE def custom_forward(*inputs): return tuple(module(*inputs, use_cache, output_attentions))
I also tried to fix a few of these issues myself but it goes deeper and there’s a lot of incompatibilities with Torchscript. Is there a better way to do this?
Thanks a lot in advance!