Generating Croissant Metadata for Custom Image Dataset

John6666 · April 15, 2025, 6:03am

services/worker/src/worker/job_runners/dataset/croissant_crumbs.py

main

# SPDX-License-Identifier: Apache-2.0
# Copyright 2024 The HuggingFace Authors.

import logging
import re
from collections.abc import Mapping
from itertools import islice
from typing import Any

from datasets import Features
from libcommon.constants import CROISSANT_MAX_CONFIGS
from libcommon.croissant_utils import feature_to_croissant_field, get_record_set
from libcommon.exceptions import PreviousStepFormatError
from libcommon.simple_cache import (
    get_previous_step_or_raise,
)

from worker.dtos import CompleteJobResult
from worker.job_runners.dataset.dataset_job_runner import DatasetJobRunner

This file has been truncated. show original

I couldn’t find documents for writing Croissant metadata manually…
There is the source code…

For now, I think I’ve figured out the order in which the bugs are occurring. The relative path is causing problems.

# FileNotFoundError: datasets/yinyue27/RefRef@main/image_data/textured_sphere_scene/single-convex/ball_sphere/./train/r_0.png

# /ball_sphere/./train/r_0.png <=There is a failure in the concatenation of the path.

Topic		Replies	Views
Dataset preview not showing for uploaded DatasetDict 🤗Datasets	6	2148	December 7, 2021
Sharing a community provided dataset Beginners	3	452	October 4, 2020
Custom loading dataset script 🤗Datasets	4	516	January 3, 2023
Uploading image dataset to Huggingface Hub 🤗Datasets	2	2615	October 14, 2022
Create Dataset with metadata 🤗Datasets	1	1378	November 28, 2022

Generating Croissant Metadata for Custom Image Dataset

Related topics