Application of a transformer model without fine tuning for NER task

Beginners
1

I am interested in using pre-trained models from Huggingface for named entity recognition (NER) tasks without any further training or testing of the model. The only information on Huggingface for the model are to use the following lines:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

I tried the following code, but I am getting a tensor output instead of class labels for each named entity.

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

text = "my text for named entity recognition here."

input_ids = torch.tensor(tokenizer.encode(text, padding=True, truncation=True,max_length=50, add_special_tokens = True)).unsqueeze(0)

with torch.no_grad():
  output = model(input_ids, output_attentions=True)

Can anyone suggest what am I doing wrong here? It will be good to have a short tutorial on how to use a pre-trained model for NER (without any fine tuning).

2

hey @shoaibb, my suggestion would be to use the ner pipeline (see docs) to extract the named entities because this is quite tricky to get right.

also, emilyalsentzer/Bio_ClinicalBERT appears to be a pretrained language model so you’ll need to either fine-tune it on a labelled NER dataset or see if you can find an existing model on the Hub that suits your needs. you can find an example on fine-tuning NER models here: Google Colaboratory

3

@lewtun Thanks a lot for the useful info.

1 Like