How to use T5 for sentence embedding?

[mriggs]

@valhalla

model = T5Model.from_pretrained("t5-small")
tok = T5Tokenizer.from_pretrained("t5-small")

enc = tok("some text", return_tensors="pt")

# forward pass through encoder only
output = model.encoder(
    input_ids=enc["input_ids"], 
    attention_mask=enc["attention_mask"], 
    return_dict=True
)
# get the final hidden states
emb = output.last_hidden_state

Hi, I’m interested in using T5 to generate word embeddings. I tried the code supplied above. Unfortunately, got this error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-40-5f6e22d1ad1e> in <module>()
      1 model = T5Model.from_pretrained("t5-small")
----> 2 tok = T5Tokenizer.from_pretrained("t5-small")
      3 
      4 enc = tok("some text", return_tensors="pt")
      5 

TypeError: 'NoneType' object is not callable

Do you have any thoughts on resolving this error message?

Thank you in advance for your help.