Model for high-dim numerical input that supports MAE?


I’m seeking advice for a transformer model to perform a regression task on high-dimensional numerical data, specifically PPG (1-D vectors) with some additional concatenated numerical features. The caveat is that I also want to perform MAE pre-training with a substantial unlabeled dataset. The closest thing I’ve stumbled across is the VitMAE model, but it’s an image classification model.

Any advice would be much appreciated, as I have limited experience working with transformer models, and then strictly in the NLP space. Thanks.