I have a strange situation where I’m trying to build a custom tokenizer for a custom “language” (encoded music). The language is designed to represent the data in a way that is compact and also reasonably human-readable. In most cases, the ideal tokenization would be essentially word-level, just sep…

Tokenized sequence lengths

jbmaxwell March 1, 2022, 4:10pm 6

Yes, that’s what I finally realized, in a kind of roundabout way.

And yes again, we should talk!