GluonTS notebook for correctly formatting Time Series Datasets for the Hub

Im going to try to keep this short and sweet. I have intraday price data for every equity in the SP500 down to M5 going back 2-3 years. The data is in CSV format. There are gaps in the time series as for when trading was closed.

I want to contribute to the hub and add all the data there for anyone to use.

I cannot for the life of me figure out how to properly format it into a dataset that makes sense. Yes, I have read all the GluonTS docs that I can find, and have been using the code posted here (notebooks/time_series_datasets.ipynb at main · huggingface/notebooks · GitHub) as reference.
Been working on this problem for over 80 hours now and I’m embarrassed to say I haven’t figured it out yet.
I understand most of the requirements of the functions in the GluonTS notebook linked above and GluonTS PandasDatasets because I’ve tripped every error message possible while trying to hack this out.

I know it should be a simple thing to do and hope someone smarter than me can help:

This is the format of the data while its in CSV.
I’ve gotten modified versions of the linked notebook to run without error, but I cant figure out a version that results in a dataset that makes sense for actually training a model.

The final dataset should have a start key entry for each row in the time series, I’ve been working on the problem trying to get o,h,l,c,v, values into the target field, this may be wrong though because likely should be predicting for next day close delta, and the other fields are just data inputs to train on, but i don’t know.

This is the output format I’m trying to make as reference.

Whoever can give me some help will be greatly appreciated. I will offer you the pick of price history data, any market, US or international, down to 5m timeframe up to 3 years back.

There’s not a lot of point in training intraday back more than five years because of regime change, but daily data is available too, at a much larger timerange.

Hi ! You can reuse the code from monash_tsf.py · monash_tsf at main

It loads data from TSF files using the functions in utils.py but you can implement a function that reads the data from CSV. Maybe @kashif can help explaining the output format of the TSF parser ?

Yes sure let me have a look and get back to you!