I get the following error: Exception has occurred: UnboundLocalError local variable 'batch_idx' referenced before assignment File "/lfs/skampere1/0/brando9/beyond-scale-language-data-diversity/src/training/utils.py", line 254, in _test_train_dataset_setup_for_main_code print(f'{len(next(iter(…

The issue happened because llama2 does not have a max sequence length set. So it defaults to max int 1000000000000000019884624838656 . You can set it manually if you google the max seq len for your model e.g., for llama2-7b: # - Get tokenized train data set # Note: Setting `batched=True` in…

Why do I get UnboundLocalError: local variable 'batch_idx' referenced before assignment when using interleaved data sets with Hugging Face (HF)?

Beginners

system Closed January 19, 2024, 11:31am 4

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Dataset.transform() hangs indefinitely while finetuning the stable diffusion XL 🤗Transformers	3	8073	January 27, 2024
How does one create a pytoch data loader using an interleaved hugging face dataset? Beginners	3	1612	August 18, 2023
ValueError: expected sequence of length 128 at dim 1 (got 68) Intermediate	0	497	October 25, 2023
I set up a different batch_size, but the time of data processing has not changed 🤗Tokenizers	0	537	September 1, 2021
Cannot Start the training loop because of bad size tokenization and/or for (presumably) custom dataset settings Beginners	2	308	June 11, 2022

Why do I get UnboundLocalError: local variable 'batch_idx' referenced before assignment when using interleaved data sets with Hugging Face (HF)?

Related topics