How to export facebook/mbart-large-50-many-to-many-mmt to TorchScript format?

Beginners
1

I’m not sure if I’m in the right place (if not, I apologize). Basically, I need help.
I am just exported pretrained to TorchScript format , using following code:

import torch
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

tokenizer = MBart50TokenizerFast.from_pretrained("d:/Install/TensorFlow/models/MBART_Base/mbart_large_50_model/", src_lang="en_XX", tgt_lang="en_XX")
example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"

expected_translation_text = "UN Chief Says There Is No Military Solution in Syria"

inputs = tokenizer(example_english_phrase, text_target=expected_translation_text, return_tensors="pt")

model_path = "d:/Install/TensorFlow/models/MBART_Base/mbart_large_50_model/"
model = MBartForConditionalGeneration.from_pretrained(model_path, torchscript=True)


traced_model = torch.jit.trace(model, [inputs.input_ids, inputs.attention_mask])
torch.jit.save(traced_model, "d:/Install/TensorFlow/models/MBART_Base/TorchScript/mbart_large_50_model_traced.pt")	

#scripted_model = torch.jit.script(model)
#torch.jit.save(scripted_model, "d:/Install/TensorFlow/models/MBART_Base/TorchScript/mbart_large_50_model_scripted.pt")

-Got something without errors, but when I tried to use resulting TorchScript format in order to translate text between languages:

import torch
from transformers import MBart50TokenizerFast

model_path = "d:/Install/TensorFlow/models/MBART_Base/TorchScript/mbart_large_50_model_traced.pt"
model = torch.jit.load(model_path)

tokenizer_path = "d:/Install/TensorFlow/models/MBART_Base/mbart_large_50_model/"
tokenizer = MBart50TokenizerFast.from_pretrained(tokenizer_path, src_lang="en_XX", tgt_lang="hr_HR")

example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"
inputs = tokenizer(example_english_phrase, return_tensors="pt")

decoded_text = tokenizer.decode(inputs['input_ids'][0], skip_special_tokens=True)

input_ids = inputs["input_ids"]

target_lang_id = tokenizer.lang_code_to_id["hr_HR"]
attention_mask = inputs["attention_mask"]
outputs = model(input_ids=input_ids, attention_mask=attention_mask)

predicted_ids = torch.argmax(outputs[0], dim=-1)
decoded_output = tokenizer.decode(predicted_ids[0], skip_special_tokens=True)
print("Translated text:", decoded_output)
  • I am getting the original input text, without first word:

Chief Says There Is No Military Solution in Syria

So, two questions:

  1. is this a correct way to export model to traced TorchScript?
  2. is this correct way to use that script?

-Any help?

1 Like
2

1. Is this the correct way to export the model to traced TorchScript?

While your approach to export the model using torch.jit.trace works without errors, torch.jit.trace is not ideal for models with dynamic control flow, such as those in transformers. MBartForConditionalGeneration has dynamic behaviors due to its attention mechanisms and token generation steps. Using torch.jit.script is generally more suitable for such models.

Here’s how you can properly export the model with scripting:

python

Copy code

from transformers import MBartForConditionalGeneration
import torch

model_path = "d:/Install/TensorFlow/models/MBART_Base/mbart_large_50_model/"
model = MBartForConditionalGeneration.from_pretrained(model_path, torchscript=True)

# Use scripting for better support of dynamic control flow
scripted_model = torch.jit.script(model)
torch.jit.save(scripted_model, "d:/Install/TensorFlow/models/MBART_Base/TorchScript/mbart_large_50_model_scripted.pt")

2. Is this the correct way to use that script?

Your usage approach has a few errors:

Key Problems in Your Usage Code:

  1. Input Preparation:
  • inputs["input_ids"] contains the tokenized input, not the decoder input IDs required for generation.
  • In MBart, decoding typically requires a start token (<bos> or language-specific start-of-sequence token). You’re not providing this explicitly.
  1. Model Output Handling:
  • torch.jit.trace restricts how the model processes inputs because of its static behavior. This might result in missing or malformed outputs.
  • You’re directly applying torch.argmax to the outputs without ensuring they’re logits for token probabilities.
  1. Decoding Issues:
  • Decoding your inputs (inputs['input_ids']) instead of the generated outputs from the model leads to displaying the original input text, minus potential encoding errors.
1 Like
3 
import torch
from transformers import MBart50TokenizerFast

# Load TorchScript model and tokenizer
model_path = "d:/Install/TensorFlow/models/MBART_Base/TorchScript/mbart_large_50_model_scripted.pt"
model = torch.jit.load(model_path)

tokenizer_path = "d:/Install/TensorFlow/models/MBART_Base/mbart_large_50_model/"
tokenizer = MBart50TokenizerFast.from_pretrained(tokenizer_path, src_lang="en_XX", tgt_lang="hr_HR")

# Prepare inputs
example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"
inputs = tokenizer(example_english_phrase, return_tensors="pt")

# Set decoder start token
decoder_start_token_id = tokenizer.lang_code_to_id["hr_HR"]

# Generate translation
outputs = model.generate(
    inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    decoder_start_token_id=decoder_start_token_id,
    max_length=50,
)

# Decode and print translation
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Translated text:", decoded_output)
1 Like
4

Hi @Alanturner2 and thanks for response!
At the line in your code:
scripted_model = torch.jit.script(model)
I am getting:

RuntimeError: 'Tensor' object has no attribute or method 'forward'.:
  File "d:\IDEJE_ZA_AI_APP\Hugging Face Modeli\pegasus-xsum\venvs\py3.9\lib\site
-packages\transformers\models\mbart\modeling_mbart.py", line 124
    def forward(self, input_ids: torch.Tensor):
        return super().forward(input_ids) * self.embed_scale
               ~~~~~~~~~~~~~ <--- HERE

-if you need any information about Python library versions I am using, I can give it to you.

1 Like
5

The error occurs because torch.jit.script attempts to compile your model into TorchScript but encounters an issue in the forward method of the MBART model you’re using. The specific problem arises because the super().forward() call in transformers.models.mbart.modeling_mbart refers to a parent class method that TorchScript cannot handle correctly.

import torch
from transformers import MBart50TokenizerFast

# Load TorchScript model
model_path = "d:/Install/TensorFlow/models/MBART_Base/TorchScript/mbart_large_50_model_scripted.pt"
model = torch.jit.load(model_path)

# Load tokenizer
tokenizer_path = "d:/Install/TensorFlow/models/MBART_Base/mbart_large_50_model/"
tokenizer = MBart50TokenizerFast.from_pretrained(tokenizer_path, src_lang="en_XX", tgt_lang="hr_HR")

# Prepare inputs
example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"
inputs = tokenizer(example_english_phrase, return_tensors="pt")

# Set decoder start token
decoder_start_token_id = tokenizer.lang_code_to_id["hr_HR"]

# Ensure inputs are TorchScript-compatible
def prepare_inputs(inputs):
    input_ids = inputs["input_ids"]
    attention_mask = inputs.get("attention_mask", None)
    return input_ids, attention_mask

input_ids, attention_mask = prepare_inputs(inputs)

# Perform inference
with torch.no_grad():
    outputs = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        decoder_start_token_id=decoder_start_token_id,
        max_length=50,
    )

# Decode and print the translation
decoded_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Translated text:", decoded_output)
2 Likes
6

HI @Alanturner2 , just to be clear:
The Runtime error I mentioned:

RuntimeError: 'Tensor' object has no attribute or method 'forward'.:

occurs when I try to generate the model in Torchscript format, so exactly in this piece of code you posted:

# Use scripting for better support of dynamic control flow
scripted_model = torch.jit.script(model)        <------ HERE
torch.jit.save(scripted_model, "d:/Install/TensorFlow/models/MBART_Base/TorchScript/mbart_large_50_model_scripted.pt")

So I am unable to obtain the scripted_model at all.
I read your explanation, and the Python script you provided afterwards assumes that there is already a generated mbart_large_50_model_scripted.pt But I was unable to create it, due to an error that occurred…huh?

1 Like
7

How about change the python library that was installed in your computer. It is difficult but you can do it. Sometimes I met the error that I couldn’t deal with, I tried to change the code of the framework or library intentionally.

1 Like
8

Thanks, I’ll try.
The thing is that I’m a Java developer, who deals with Python only as much as necessary, and I know it poorly. Namely, I need the export to this format for only one reason - to get a format that I can work with using Java libraries.

Also, have you (or anyone else) tried to export this model to TorchScript?
Since I’m still not sure if I’m doing something wrong, or if there’s a bug in the Python implementation…

1 Like
9

I know what you want to know.
During my developing, there is sometimes so many problems. And I had to use framework that I didn’t know. At that time,
First, I tried to read and scan the code of the python lib and especially use dir() function. dir()function is very useful to know the structure or parameters of the function or object.
And next, I also use GPT. That is very helpful and powerful in python code. It knows only old versions but they are similar. If there isn’t in GPT, then I tried to read the documentation.
I hope to help your coding.

2 Likes