Hi everyone, I want to train a self-supervised Bert model using masked language modelling, but my task involves sequential data with a mix of numerical and categorical features. While I know that categorical data is encoded using Embedding layers, I wanted to know if there is any library implementation for transforming numerical features into the latent space, which will then be fed in as input.
Maybe something similar to this paper: 2203.05556.pdf (arxiv.org)