Hi everyone! I’d really appreciate your help with an issue I’m having with BERT for NER in a highly specialized domain. I’m using BERT for token classification and used much of the format from here (Named entity recognition with Bert) as inspiration. I had to generate my own B/I labeling for my dataset, which looks fine - some samples are below. Text lengths were also often longer than BERT max tokens of 512 so I had a chunk a given text piece into smaller chunks (of which at the start I’d place a [CLS] token. I didn’t really add [SEP] tokens on any of these chunked pieces as they’d have padding to reach the length of 512. The labels, input id’s, padding and attention masks appear fine and I’ve also pasted some samples below. I’ve also checked the output numerous times - again my code is below for the predict function. I had to aggregate the byte level predictions to come up with labels at the original word level and did this by averaging over the byte level logit outputs if a word was split by the tokenzier (this is done by the aggregate function). Another idiosyncratic part of my predict function is I had to know indices in the original text that correspond to tokens - the first part of the predict function is just storing those values for the original ‘words’. What’s weird is if I train the BERT model on a handful of data samples (for instance, <10), I’ll get some output that has none ‘0’ predictions. Training over a larger set seems to result in only ‘0’ predictions. Really don’t understand what’s going on at all and would appreciate any help - thank you so much!
[… ‘P’, ‘##H’, ‘##Y’, ‘##SI’, ‘##CI’, ‘##AN’, ‘IN’, ‘##TE’, ‘##RP’, ‘##RE’, ‘##TA’, ‘##TI’, ‘##ON’, ‘:’, ‘N’, ‘##eg’, ‘##ative’, ‘:’, ‘Tu’, ‘##mor’, ‘cells’, ‘showing’, ‘no’]
[…‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘B-HER2’, ‘B-HER2’, ‘B-HER2’, ‘B-HER2’, ‘I-HER2’, ‘I-HER2’, ‘I-HER2’, ‘I-HER2’, ‘I-HER2’]
def predict(test_texts, test_note_ids, model, tokenizer):
offset = 0
model.eval()
for test_text, test_note_id in zip(test_texts, test_note_ids):
num_notes += 1
chunkified_tokenized_texts, _, _ = chunkify_note(test_text, None, test_note_id)
predictions = []
for split_text in chunkified_tokenized_texts:
tokenized_test_text = tokenizer.encode(split_text)
tok_list = []
for m in re.finditer(r'\S+', split_text):
token = m.group(0)
sub_tokens = tokenizer.tokenize(token)
tok_list.append((token, offset+m.start(), offset+m.end() - 1))
offset = len(split_text)
if torch.cuda.is_available():
input_ids = torch.tensor([tokenized_test_text]).cuda()
else:
input_ids = torch.tensor([tokenized_test_text])
with torch.no_grad():
output = model(input_ids)
if torch.cuda.is_available():
scores = output[0].detach().cpu().numpy()
else:
scores = output[0].detach().numpy()
pred = aggregate(scores, tokenizer, tok_list)
predictions.extend(pred) #then do some extra processing to store this
Input Id’s on the input:
[ 101 1239 1104 115 115 115 115 115 117 115 115 115
115 115 115 115 115 115 115 4183 2524 115 115 115
115 115 115 115 115 115 115 115 115 115 115 115
117 115 115 115 115 115 115 115 115 115 115 115
115 115 115 115 115 115 115 115 115 117 9292 117
7735 115 115 115 115 115 115 115 115 115 115 117
115 115 115 115 115 115 115 115 115 115 118 115
115 115 115 115 115 115 115 115 115 115 115 115
115 115 2524 11341 131 113 115 115 115 115 115 114
115 115 115 115 115 118 115 115 115 115 115 115
115 115 115 115 115 115 115 115 115 119 115 115
115 115 115 117 9292 117 7735 143 7897 131 113 115
115 115 115 115 114 115 115 115 115 115 118 115
115 115 115 115 9666 9970 18653 11922 131 115 115 115
115 115 137 115 115 115 115 115 119 115 115 115
115 115 115 115 115 115 115 115 115 115 115 115
115 115 115 115 115 117 7735 115 115 115 115 115
115 115 115 115 115 117 9292 117 7735 150 13901 8231
2591 10783 2069 8544 24162 13901 2346 2349 3663 155 16668 9565
1942 7195 9080 10208 131 115 115 115 115 115 117 115
115 115 115 115 115 115 115 115 115 108 131 115
115 115 115 115 118 115 115 115 115 115 2508 1181
119 11336 1665 119 108 131 115 115 115 115 115 159
26868 1204 108 131 115 115 115 115 115 2516 14265 131
1429 120 1407 120 1446 26316 131 1492 2162 9850 131 9714
11336 21437 131 1367 120 5037 120 1446 140 19526 131 115
115 115 115 115 141 2346 2064 131 1429 120 1429 120
2679 113 4936 131 4062 114 7642 6834 27989 1389 113 188
114 131 115 115 115 115 115 115 115 115 115 115
119 115 115 115 115 115 113 113 115 115 115 115
115 114 115 115 115 115 115 118 115 115 115 115
115 114 115 115 115 115 115 115 115 115 115 115
113 113 115 115 115 115 115 114 115 115 115 115
115 118 115 115 115 115 115 114 2687 7136 6902 131
15075 10584 1179 118 4275 117 18311 16274 118 11783 7918 1113
2525 15766 119 115 115 115 115 115 118 115 115 115
115 115 115 115 115 115 115 131 4114 7209 15961 145
1775 131 115 115 115 115 115 119 5539 1580 115 115
115 115 115 115 115 115 115 115 7277 1643 17489 27259
1118 143 6258 3048 10722 23904 9664 18082 2069 3663 153 3048
3663 13882 19747 14962 15969 12880 20336 16941 9159 21669 11414 131
151 12606 5838 131 17037 26271 3652 4000 1185 0 0 0
0 0 0 0 0 0 0 0]
Tagged values:
[ 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
9 9 9 9 9 9 9 9 9 9 9 9 6 6 6 6 4 4 4 4 4 27 27 27
27 27 27 27 27 27 27 27]
Attention mask:
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]