|
Customization of Wav2Vec2CTCTokenizer with rules
|
|
0
|
126
|
August 22, 2022
|
|
Customized tokenization files in run_clm script
|
|
3
|
143
|
August 18, 2022
|
|
Using customized algorithm
|
|
0
|
127
|
August 17, 2022
|
|
Issue with Flaubert Tokenizer as word_ids() method is not available for NER Task
|
|
1
|
246
|
August 15, 2022
|
|
NER Label tokenization with overflowing tokens
|
|
0
|
109
|
August 12, 2022
|
|
Word_ids not working with deberta_v2
|
|
1
|
201
|
August 12, 2022
|
|
How to tokenize large contexts without running out of memory
|
|
2
|
846
|
August 8, 2022
|
|
Does Deberta tokenizer use wordpiece?
|
|
0
|
200
|
August 6, 2022
|
|
"Add_tokens" breaks words when encoding
|
|
0
|
187
|
August 2, 2022
|
|
Get vocabulary tokens in order to exclude them from generate function
|
|
2
|
1267
|
August 1, 2022
|
|
Avoid creating certain tokens when training a tokenizer
|
|
0
|
194
|
July 26, 2022
|
|
Remove only certain special token id during tokenizer decode
|
|
0
|
210
|
July 15, 2022
|
|
Error finetuning XLM-RoBERTa-Large when training
|
|
2
|
220
|
July 15, 2022
|
|
HuggingFace BPE Trainer Error - Training Tokenizer
|
|
1
|
927
|
July 14, 2022
|
|
Word_to_tokens() and word_ids() ---- microsoft/deberta-v2/v3
|
|
2
|
262
|
July 14, 2022
|
|
No PreTrainedTokenizerFast for Deberta-V3, no doc_stride
|
|
0
|
211
|
July 13, 2022
|
|
Tokenizer from own vocab
|
|
0
|
149
|
July 11, 2022
|
|
Tokenizer not recognising words in vocabulary
|
|
0
|
219
|
July 8, 2022
|
|
Tokenizer dataset is very slow
|
|
1
|
315
|
June 28, 2022
|
|
No labels column for tokenized data
|
|
2
|
314
|
June 27, 2022
|
|
Programmatic way to Tokenization on Custom Text Columns
|
|
0
|
271
|
June 27, 2022
|
|
Bug in Offset generation for Rupee symbol
|
|
0
|
251
|
June 27, 2022
|
|
How to handle parenthesis, quotation marks, \n etc when creating tokenizer from scratch
|
|
0
|
274
|
June 26, 2022
|
|
EM training on unigram tokenizer taking way longer than predicted
|
|
0
|
232
|
June 23, 2022
|
|
Training unigram on long sequences
|
|
4
|
405
|
June 23, 2022
|
|
Sliding window for Long Documents
|
|
0
|
299
|
June 20, 2022
|
|
Issue with post-processing
|
|
1
|
658
|
June 15, 2022
|
|
FutureWarning about BertTokenizer.from_pretrained() at latest version
|
|
0
|
378
|
June 6, 2022
|
|
Enhaced word_ids() API for Chinese or CJK languages?
|
|
0
|
333
|
June 2, 2022
|
|
Importing tokenizers version >0.10.3 fails due to openssl
|
|
3
|
1470
|
June 2, 2022
|