Generate() method for models converted to torchscript

Models
1

I’ve been experimenting exporting git-large-coco to torchscript and with a minor adjustment to the transformers library this seems to work. However GitForCausalLM uses the generate() function, not just just a plain model() invocation, so I’m stuck on how to use the torchscript version of the model.

Am I correct that I would need to reimplement whatever generate() is doing if I want to use the torchscript model? I’d eventually like to run it in a non-python environment, but for now I’m only experimenting.

(edit: I did find Generate without using the generate method which give me some clues, but I’m not getting the correct output yet)

Can I also load the torchscript model back into transformers just to see if the model behaves the same?

2

Were you able to make any progress on this? I tried implementing the code from the post you have linked. But it is much slower than the generate( ) method.

3

No not really. I did try to implement it but I think the model I’m using is not using greedy matching or something and the results weren’t similar. Still not sure why though.

Then I found out I messed up the torchscript conversion and that it’s probably not going to work for what I’m trying to do, so I didn’t pursue it further.