Hi,
I want to add an image dataset that requires a custom script.
What is the recommended way of handling images in _generate_example in a custom script?
Based on the docs (link) a few approaches are allowed: absolute path (alone), np.ndarray, PIL.Image.Image, dict with relate path and bytes {"path": ..., "bytes": ...}.
I saw a few different approaches applied to different datasets in the Hub. MNIST reads all the objects (images and labels) and then yields np.ndarray (I guess the aspect of reading all files prior to using yield won’t scale) (same approach in fashion_mnist). imagenette yields the dict with path and bytes. alkzar90/NIH-Chest-X-ray-dataset returns paths.
Which of these approaches gives the best performance, or are there any inclinations to choose one over another? Drawbacks? Pros? Should I pay special attention to anything? (I can maintain all of them etc. I’m interested in the best performance)