Other (better) solution is to store the data in a format that does not require to upload each file individually. Typically, it could .parquet files with columns like “date” “filename” “content” where each row is a markdown file. This way you will have only a few .parquet files to upload which will solve all of your problems. Also, it will enable the Dataset Studio for your repo.
2 Likes