Hello @nielsr and @kashif,
New to time series task here. I am following the Time Series Transformer tutorial and have managed to put a time-series dataset in the huggingface convention but the dataset is not split; it is singular.
From my understanding of the test dataset in your notebook is actually just the train dataset + the future values we want the model to forecast. Meaning the train is actually duplicated to create the test set.
Is it possible to create the test set without duplicating the entire train set. My dataset is huge 13M rows and 32 columns.
so we need a copy of the full time series in the test split since the context-window is a user-defined param. and thus one would need to load the very last chunk of this from the train split… and so in that case it was easier to just have the full time series with prediction length of more data in the test split. I know that this results in data duplication…
1 Like