Can anyone help me how to split the dataset into training and test like the way it was defined earlier-:
# load dataset
train_dataset, test_dataset = load_dataset(dataset_name, split=[‘train’, ‘test’])
This is not working and i am using b-mc2/sql-create-context dataset and need to split it into train and test.
Hi! b-mc2/sql-create-context has a single train split (built from a single JSON file), so you need to use Dataset.train_test_split to achieve that:
train_dataset = load_dataset("b-mc2/sql-create-context ", split="train")
train_test_dataset = train_dataset.train_test_split(test_size=0.2)
train_dataset, test_dataset = train_test_dataset["train"], train_test_dataset["test"]