Hi, I have a conll-format dataset where the training “split” comprises multiple files. I’m writing a loader based on the conll2003 loader but can’t figure out what the best way is to load in multiple files for a single split. SplitGenerator seems to pass just a single filepath on in gen_kwargs param…

Loader for dataset with multiple source files in one split

lhoestq May 9, 2022, 10:06am 2

Hi ! You can just pass a list of files to gen_kwargs, and iterate over the files in _generate_examples

1 Like