I have fine tuned longformer model on LongformerForMaskedLM and saved it as .bin file.
I have loaded the bin file and I want to extract sentence embedding for sentences ( Normalised by attention masks).
Here is my code:
model = LongformerModel.from_pretrained('checkpoint_longformer',output_hidden_states = True)
tokenizer = LongformerTokenizer.from_pretrained('checkpoint_longformer')
# Put the model in "evaluation" mode, meaning feed-forward operation.
model.eval()
data=pd.read_csv("sentences.csv")
all_content=list(data['product_review'])
From here how should I get sentence/document level embedding for every row in batches. ( Normalised by attention masks).
a
I am stuck at this point. Please help.