Hello everyone, I’ve been referencing this paper on training transformer based models using metadata enhanced MIDI and was thinking about implementing this using the huggingface transformers and tokenizer libraries as an introduction to these libraries beyond the basic language modeling examples. As I’ve been researching and referencing this tutorial I’ve ran into issues with tokenization and was wondering when training a tokenizer, how can I set up “word level” semantics? Technically each “word” in this case will be the data within a string like so ‘Event(name=Position, time=360, value=4/16, text=360)’ rather than just words and characters delimited on spaces like its doing now as listed below
#version: 0.2 - Trained by huggingface/tokenizers
Apologies on if these questions are noobish I’m grokking a lot of this as I go along. Any help is greatly appreciated.