Opcodeo Tokenizer

natankatz May 17, 2023, 6:24am 1

Is there a tokenizer for opcode sources? (a model will be even better)

Topic	Replies	Views
ESM Fast Tokenizer Beginners	77	January 15, 2024
Skip-gram tokens 🤗Tokenizers	339	April 4, 2021
Custom Tokenizer for source code Beginners	379	March 4, 2022
Tokenizers v0.8.0 is out! 🤗Tokenizers	1409	July 7, 2020
Easy way to compare tokenizers 🤗Tokenizers	243	May 1, 2023