Is there a tokenizer for opcode sources? (a model will be even better)
Related topics
Topic | Replies | Views | Activity | |
---|---|---|---|---|
ESM Fast Tokenizer | 0 | 135 | January 15, 2024 | |
Skip-gram tokens | 0 | 370 | April 4, 2021 | |
Rs-bpe tokenizer [PyPI | Python] - Outperforms tiktoken & tokenizers | 2 | 46 | March 19, 2025 | |
Rs-bpe [PyPI | Python] - Outperforms tiktoken & tokenizers | 1 | 25 | March 20, 2025 | |
Custom Tokenizer for source code | 0 | 441 | March 4, 2022 |