Do you have an example that doesn’t use trust_remote_code
? We stopped developing that option because it’s not great for obvious security reasons.
Also note that datasets in WebDataset format are supported out-of-the-box so trust_remote_code
/ having a loading script is not needed in that case.