How to sample from the validation set when using Trainer?

[ruan-xf]

Time has passed, but for people coming here:

that’s what I want for my large eval dataset, and I totally got you, it means fast and frequent evaluation but still make use of the existing eval dataset. it’s possible with TrainerCallback and indefinite IterableDataset. I’ve replied here:

Evaluate subset of data during training

I implement a “sliding” eval dataset by using TrainerCallback and indefinite IterableDataset, here it’s: class DatasetSetting: def __init__(self, do_encode: bool, per_eval_size: int, process_fn: Callable=None): user_enc, item_enc = None, None if do_encode: pkl_path = Path('data/user_item_encoder.pkl') if not pkl_path.exists(): fit_encoder() enc: OrdinalEncoder = joblib.load(pkl_path) user_enc, item_enc = (MyEncoder(cate) for c…