Converting TikToken to Huggingface Tokenizer
|
|
0
|
1747
|
March 10, 2023
|
ArrowInvalid: Column 3 named attention_mask expected length 1000 but got length 1076
|
|
3
|
2352
|
July 26, 2023
|
Discussing the Pros and Cons of Using add_tokens vs. Byte Pair Encoding (BPE) for Adding New Tokens to an Existing RoBERTa Model
|
|
0
|
501
|
July 14, 2023
|
Initialize Vocabulary for Unigram Tokenizer
|
|
0
|
229
|
July 11, 2023
|
Make correct padding for text generation with GPT-NEO
|
|
0
|
565
|
July 5, 2023
|
How does a tokenzier (eg., AutoTokenizer) generate word_ids intergers?
|
|
0
|
403
|
June 26, 2023
|
Seeking an end-to-end example of grouping, tokenization and padding to construct preprocessed data in HF
|
|
0
|
315
|
June 26, 2023
|
Writing custom tokenizer and wrapping it in tokenizer object
|
|
2
|
517
|
June 26, 2023
|
Tokenizer for German lang
|
|
0
|
332
|
June 22, 2023
|
Chunk tokens into desired chunk length without simply getting rid of rest of tokens
|
|
0
|
452
|
June 15, 2023
|
Padding not transferring when loading a tokenizer trained via the tokenizers library into transformers
|
|
0
|
365
|
June 12, 2023
|
LlamaTokenizerFast returns token_type_ids but the forward pass of the LlamaModel does not receive token_type_ids
|
|
1
|
549
|
June 9, 2023
|
GPT2Tokenizer not working in Kaggle Notebook
|
|
0
|
278
|
May 30, 2023
|
Exploring the Majestic Temples in Karnataka
|
|
0
|
258
|
May 25, 2023
|
Tokenizer producing token index greater than size of the dictionary
|
|
0
|
235
|
May 15, 2023
|
How to instantiate a XLMRobertaTokenizer object using a locally trained SentencePiece tokenizer
|
|
0
|
230
|
May 14, 2023
|
How to create a HF tokenizer's vocab file from a BPE model's merges.txt file?
|
|
0
|
352
|
May 13, 2023
|
Scala/JVM Bindings for Tokenizers
|
|
0
|
306
|
May 10, 2023
|
Tokenizers Wheel Takes Forever to Build
|
|
1
|
2603
|
May 8, 2023
|
Where the introduction of tokenizers.implementations?
|
|
0
|
154
|
May 7, 2023
|
How to return custom `token_type_ids` or other values from a tokenizer?
|
|
0
|
451
|
May 3, 2023
|
Easy way to compare tokenizers
|
|
0
|
220
|
May 1, 2023
|
Issue with XLM-RoBERTa tokenizer
|
|
0
|
220
|
May 1, 2023
|
Unable to load image using llama-index
|
|
0
|
1088
|
May 1, 2023
|
Help defining tokenizer
|
|
0
|
203
|
April 28, 2023
|
Token Offsets in Rust vs. Python
|
|
1
|
239
|
April 27, 2023
|
āOSError: Model name './XX' was not found in tokenizers model name listā - cannot load custom tokenizer in Transformers
|
|
14
|
6312
|
April 25, 2023
|
Converting JSON/dict to flatten string with indicator tokens
|
|
1
|
282
|
April 21, 2023
|
Train Retry Tokenizer
|
|
0
|
198
|
April 18, 2023
|
Pretokenise on punctuation except hyphens
|
|
0
|
220
|
April 15, 2023
|