Hi,
I see HF Models are versioned, and HF Datasets are versioned too.
Hi Olivier,
in the model metadata spec, you’ll see we suggest encoding the version (= git-revision) of the dataset inside the model-index dict, i.E. the place where you encode your eval results in a structured way:
- name: {model_id}
results:
- task:
type: {task_type} # Required. Example: automatic-speech-recognition
name: {task_name} # Optional. Example: Speech Recognition
dataset:
type: {dataset_type} # Required. Example: common_voice. Use dataset id from https://hf.co/datasets
name: {dataset_name} # Required. A pretty name for the dataset. Example: Common Voice (French)
config: {dataset_config} # Optional. The name of the dataset configuration used in `load_dataset()`. Example: fr in `load_dataset("common_voice", "fr")`. See the `datasets` docs for more info: https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset.name
split: {dataset_split} # Optional. Example: test
revision: {dataset_revision} # Optional. Example: 5503434ddd753f426f4b38109466949a1217c2bb
args:
{arg_0}: {value_0} # Optional. Additional arguments to `load_dataset()`. Example for wikipedia: language: en
{arg_1}: {value_1} # Optional. Example for wikipedia: date: 20220301
metrics:
- type: {metric_type} # Required. Example: wer. Use metric id from https://hf.co/metrics
value: {metric_value} # Required. Example: 20.90
name: {metric_name} # Optional. Example: Test WER
config: {metric_config} # Optional. The name of the metric configuration used in `load_metric()`. Example: bleurt-large-512 in `load_metric("bleurt", "bleurt-large-512")`. See the `datasets` docs for more info: https://huggingface.co/docs/datasets/v2.1.0/en/loading#load-configurations
args:
{arg_0}: {value_0} # Optional. The arguments passed during `Metric.compute()`. Example for `bleu`: max_order: 4
models that are pushed by the hf trainer should include that metadata most of the time
cc @sgugger @osanseviero
→ Does the HF Trainer automatically copy the git hash of the training dataset (if it detects it’s an HF Dataset) to the model metadata?
thanks
We don’t as I’m not sure we have that information in the dataset metadata (maybe it was added recently in which case we can make a PR to add support for this).
yes i think that should be possible, cc @lhoestq
Yup that will definitely be possible, there’s an issue on this here but we haven’t started working on this yet: dataset metadata for reproducibility · Issue #4129 · huggingface/datasets · GitHub Contributions are welcome ^^