Hi,
I have been using the DataViewer to analyze my model’s output, which is pretty convenient and straightforward.
But recently, due to the dataset keeping getting bigger and bigger, for some subset, the dataviewer can only show a preview of the data, which I found explained here
Does it mean I have to manually delete some of that data to unlock the full data viewer again?
It’s reasonable to set that limit, but I wonder if there is a way that we can flexibly configure which subset to be in the preview-only mode and vice versa.
Thanks!
1 Like
This is known only to the staff… @lhoestq
Hi ! The Viewer allows to explore arbitrarily large datasets in Parquet formats. This means you can paginate through the full dataset in the Viewer. Datasets in other formats are auto-converted to Parquet up to 5GB, and in that case the Viewer only shows the data up to 5GB.
PS: this is valid for pagination, but note that the statistics as well as filter and search features are always limited to 5GB, no matter the dataset format.
1 Like
Thanks for the quick response!
Is this 5GB refer to one single dataset config (subset) or the overall dataset size?
1 Like