Trying to Build Datasets, Random Items Get Added

[muellerzr]

@sgugger

I think it’s because the “text” loader creates a new sample for each “\n” it sees, so the texts you have that contain some of those are then split into several samples. @lhoestq or @albertvillanova if you could conifrm?

I don’t think so, when I tried checking for that I still got 25,000. Or a better way to put that is this returns zero:

from fastcore.xtras import open_file

count = 0
for text in texts:
    t = open_file(text).read()
    if '\n' in t or '\r' in t: count += 1

@sgugger

PS: it would be easier to just do dsets = load_dataset('imdb') :-p

Of course it would be! However I’m trying to write a high-level data API for adaptnlp currently, so I’m only using IMDB as a situational test case

Edit: Trying a new way to verify, will update with those results

Aha! @sgugger thank you! There were some hidden \x85 characters, which is the source of the breakage.

I can work with that now. Thank you!
(If you have recommendations for fixes, I’m all ears, I was just going to take that into account while mapping labels from folder names)