Using time series for SequenceClassification models

Im thinking of using Transformer models to classify other sequential data, namely time series data.
My idea is to feed fixed-sized sequences of time series value as input into a BERT-like model with a classification head. Since using pre-trained models probably makes no sense, I would train it from scratch. Since time series values are already numerical, am I right to think that tokenization isn’t needed? How can I assure that a BERT-kind model even understands the input without using the corresponding tokenizer? Is there anything else to know when wanting to control the classification head layers, apart from passing num_values?

What steps would I undergo for this task of time series classification? Any tips? Im grateful for any ideas. Perhaps someone already knows a repository/model? That would be extremly helpful.

you might find some pointers here