Model with Genrate method to torchscript

[John6666]

There are also means such as the ONNX and GGUF formats.

Generate without using the generate method

Posting this here for visibility. What if you want to decode the output of a generative seq2seq model (like T5, BART, etc.) yourself, without using the .generate() method? The code example below illustrates this. Suppose that the model is given a long text, for which it needs to generate a summary. We illustrate here how to manually decode the generated ids autoregressively. In each iteration, we add the predicted token id by the model to the decoder_input_ids, which are then fed as input to th…

How to convert your PyTorch model to TorchScript

Common AI Model Formats