Iâve tried to fine tune multiple models using many different datasets and once i click the start training button it turns red for a couple of seconds then turns blue again, Iâve tried this with multiple models and different datasets but nothing works, Iâve included the log file below
Blockquote
Device 0: Tesla T4 - 7072MiB/15360MiB
You are not running the flash-attention implementation, expect numerical differences.
INFO | 2024-06-27 10:32:12 | autotrain.trainers.common:on_train_begin:231 - Starting to trainâŚ
Generating train split: 4262 examples [00:09, 442.90 examples/s]
Generating train split: 4000 examples [00:09, 494.18 examples/s]
Generating train split: 3723 examples [00:09, 407.22 examples/s]
Generating train split: 3262 examples [00:07, 629.96 examples/s]
Generating train split: 2843 examples [00:07, 475.21 examples/s]
Generating train split: 2245 examples [00:07, 281.14 examples/s]
Generating train split: 1415 examples [00:04, 315.25 examples/s]
Generating train split: 1000 examples [00:02, 654.30 examples/s]
Generating train split: 601 examples [00:02, 382.11 examples/s]
Generating train split: 1 examples [00:02, 2.10s/ examples]
Token indices sequence length is longer than the specified maximum sequence length for this model (2901 > 2048). Running this sequence through the model will result in indexing errors
warnings.warn(
/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:307: UserWarning: You passed a dataset_text_field argument to the SFTTrainer, the value you passed will override the one in the SFTConfig.
warnings.warn(
/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:269: UserWarning: You passed a max_seq_length argument to the SFTTrainer, the value you passed will override the one in the SFTConfig.
warnings.warn(
/app/env/lib/python3.10/site-packages/transformers/training_args.py:1965: FutureWarning: --push_to_hub_token is deprecated and will be removed in version 5 of 
Transformers. Use --hub_token instead.
warnings.warn(
/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:181: UserWarning: You passed a packing argument to the SFTTrainer, the value you passed will override the one in the SFTConfig.
warnings.warn(
/app/env/lib/python3.10/site-packages/transformers/training_args.py:1965: FutureWarning: --push_to_hub_token is deprecated and will be removed in version 5 of Transformers. Use --hub_token instead.
warnings.warn(message, FutureWarning)
Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.
/app/env/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in âinitâ: dataset_text_field, max_seq_length, packing. Will not be supported from version â1.0.0â.
INFO | 2024-06-27 10:32:01 | autotrain.trainers.clm.train_clm_sft:train:37 - creating trainer
INFO | 2024-06-27 10:32:01 | autotrain.trainers.clm.utils:get_model:666 - model dtype: torch.float16
Loading checkpoint shards: 100%|ââââââââââ| 2/2 [00:12<00:00, 6.23s/it]
Loading checkpoint shards: 100%|ââââââââââ| 2/2 [00:12<00:00, 5.95s/it]
low_cpu_mem_usage was None, now set to True since model is quantized.
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:get_model:635 - loading modelâŚ
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:get_model:627 - loading model configâŚ
WARNING | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:get_model:625 - Unsloth not available, continuing without itâŚ
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:get_model:583 - Can use unsloth: False
warnings.warn(
/app/env/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:configure_block_size:548 - Using block size 1024
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:configure_training_args:485 - configuring training args
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:configure_logging_steps:480 - Logging steps: 25
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:configure_logging_steps:467 - configuring logging steps
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:process_input_data:395 - Valid data: None
})
num_rows: 9846
features: [âtextâ],
INFO | 2024-06-27 10:31:48 | autotrain.trainers.clm.utils:process_input_data:394 - Train data: Dataset({
Repo card metadata block was not found. Setting CardData to empty.
INFO | 2024-06-27 10:31:47 | autotrain.trainers.clm.train_clm_sft:train:12 - Starting SFT trainingâŚ
e[93m [WARNING] e[0m using untested triton version (2.3.0), only 1.0.0 is known to be compatible
e[93m [WARNING] e[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
e[93m [WARNING] e[0m NVIDIA Inference is only supported on Ampere and newer architectures
e[93m [WARNING] e[0m Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
e[93m [WARNING] e[0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
e[93m [WARNING] e[0m async_io: please install the libaio-dev package with apt
e[93m [WARNING] e[0m async_io requires the dev libaio .so object and headers but these were not found.
[2024-06-27 10:31:47,544] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
--dynamo_backend was set to a value of 'no'
The following values were not passed to accelerate launch and had defaults used instead:
INFO | 2024-06-27 10:31:40 | autotrain.backends.local:create:13 - Training PID: 620
INFO | 2024-06-27 10:31:40 | autotrain.commands:launch_command:401 - {âmodelâ: âmicrosoft/Phi-3-mini-4k-instructâ, âproject_nameâ: âautotrain-uh8dc-qv9ttâ, âdata_pathâ: âtimdettmers/openassistant-guanacoâ, âtrain_splitâ: âtrainâ, âvalid_splitâ: None, âadd_eos_tokenâ: True, âblock_sizeâ: 1024, âmodel_max_lengthâ: 2048, âpaddingâ: ârightâ, âtrainerâ: âsftâ, âuse_flash_attention_2â: False, âlogâ: âtensorboardâ, âdisable_gradient_checkpointingâ: False, âlogging_stepsâ: -1, âeval_strategyâ: âepochâ, âsave_total_limitâ: 1, âauto_find_batch_sizeâ: False, âmixed_precisionâ: âfp16â, âlrâ: 3e-05, âepochsâ: 3, âbatch_sizeâ: 2, âwarmup_ratioâ: 0.1, âgradient_accumulationâ: 4, âoptimizerâ: âadamw_torchâ, âschedulerâ: âlinearâ, âweight_decayâ: 0.0, âmax_grad_normâ: 1.0, âseedâ: 42, âchat_templateâ: ânoneâ, âquantizationâ: âint4â, âtarget_modulesâ: âall-linearâ, âmerge_adapterâ: False, âpeftâ: True, âlora_râ: 16, âlora_alphaâ: 32, âlora_dropoutâ: 0.05, âmodel_refâ: None, âdpo_betaâ: 0.1, âmax_prompt_lengthâ: 128, âmax_completion_lengthâ: None, âprompt_text_columnâ: âpromptâ, âtext_columnâ: âtextâ, ârejected_text_columnâ: ârejected_textâ, âpush_to_hubâ: True, âusernameâ: âhuggingfacepremiumâ, âtokenâ: â*****â, âunslothâ: False}
INFO | 2024-06-27 10:31:40 | autotrain.commands:launch_command:400 - [âaccelerateâ, âlaunchâ, âânum_machinesâ, â1â, âânum_processesâ, â1â, ââmixed_precisionâ, âfp16â, â-mâ, âautotrain.trainers.clmâ, ââtraining_configâ, âautotrain-uh8dc-qv9tt/training_params.jsonâ]
INFO | 2024-06-27 10:31:40 | autotrain.backends.local:create:8 - Starting local trainingâŚ
INFO | 2024-06-27 10:31:40 | autotrain.app.ui_routes:handle_form:491 - hardware: local-ui
INFO | 2024-06-27 10:31:22 | autotrain.app.utils:kill_process_by_pid:52 - Sent SIGTERM to process with PID 439
INFO | 2024-06-27 10:31:22 | autotrain.app.utils:get_running_jobs:26 - Killing PID: 439
ERROR | 2024-06-27 10:31:20 | autotrain.trainers.common:wrapper:121 - Blockwise quantization only supports 16/32-bit floats, but got torch.uint8
ValueError: Blockwise quantization only supports 16/32-bit floats, but got torch.uint8
raise ValueError(f"Blockwise quantization only supports 16/32-bit floats, but got {A.dtype}")
File â/app/env/lib/python3.10/site-packages/bitsandbytes/functional.pyâ, line 1234, in quantize_4bit
w_4bit, quant_state = bnb.functional.quantize_4bit(
File â/app/env/lib/python3.10/site-packages/bitsandbytes/nn/modules.pyâ, line 289, in _quantize
return self._quantize(device)
File â/app/env/lib/python3.10/site-packages/bitsandbytes/nn/modules.pyâ, line 324, in to
new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).to(target_device)
File â/app/env/lib/python3.10/site-packages/transformers/quantizers/quantizer_bnb_4bit.pyâ, line 216, in create_quantized_param
hf_quantizer.create_quantized_param(model, param, param_name, param_device, state_dict, unexpected_keys)
File â/app/env/lib/python3.10/site-packages/transformers/modeling_utils.pyâ, line 889, in _load_state_dict_into_meta_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
File â/app/env/lib/python3.10/site-packages/transformers/modeling_utils.pyâ, line 4214, in _load_pretrained_model
) = cls._load_pretrained_model(
File â/app/env/lib/python3.10/site-packages/transformers/modeling_utils.pyâ, line 3754, in from_pretrained
return model_class.from_pretrained(
File â/app/env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.pyâ, line 563, in from_pretrained
model = AutoModelForCausalLM.from_pretrained(
File â/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/utils.pyâ, line 649, in get_model
model = utils.get_model(config, tokenizer)
File â/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/train_clm_sft.pyâ, line 25, in train
train_sft(config)
File â/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/main.pyâ, line 28, in train
return func(*args, **kwargs)
File â/app/env/lib/python3.10/site-packages/autotrain/trainers/common.pyâ, line 117, in wrapper
ERROR | 2024-06-27 10:31:20 | autotrain.trainers.common:wrapper:120 - train has failed due to an exception: Traceback (most recent call last):
low_cpu_mem_usage was None, now set to True since model is quantized.
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:get_model:635 - loading modelâŚ
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:get_model:627 - loading model configâŚ
WARNING | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:get_model:625 - Unsloth not available, continuing without itâŚ
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:get_model:583 - Can use unsloth: False
warnings.warn(
/app/env/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:configure_block_size:548 - Using block size 1024
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:configure_training_args:485 - configuring training args
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:configure_logging_steps:480 - Logging steps: 25
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:configure_logging_steps:467 - configuring logging steps
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:process_input_data:395 - Valid data: None
})
num_rows: 9846
features: [âtextâ],
INFO | 2024-06-27 10:31:13 | autotrain.trainers.clm.utils:process_input_data:394 - Train data: Dataset({
Generating test split: 100%|ââââââââââ| 518/518 [00:00<00:00, 78384.06 examples/s]
Generating test split: 0%| | 0/518 [00:00<?, ? examples/s]
Generating train split: 100%|ââââââââââ| 9846/9846 [00:00<00:00, 75561.98 examples/s]
Generating train split: 100%|ââââââââââ| 9846/9846 [00:00<00:00, 76628.26 examples/s]
Generating train split: 0%| | 0/9846 [00:00<?, ? examples/s]
Downloading data: 100%|ââââââââââ| 1.11M/1.11M [00:00<00:00, 26.2MB/s]
Downloading data: 0%| | 0.00/1.11M [00:00<?, ?B/s]
Downloading data: 100%|ââââââââââ| 20.9M/20.9M [00:01<00:00, 19.9MB/s]
Downloading data: 100%|ââââââââââ| 20.9M/20.9M [00:01<00:00, 18.9MB/s]
Downloading data: 0%| | 0.00/20.9M [00:00<?, ?B/s]
Repo card metadata block was not found. Setting CardData to empty.
Downloading readme: 100%|ââââââââââ| 395/395 [00:00<00:00, 3.05MB/s]
Downloading readme: 0%| | 0.00/395 [00:00<?, ?B/s]
INFO | 2024-06-27 10:31:10 | autotrain.trainers.clm.train_clm_sft:train:12 - Starting SFT trainingâŚ
e[93m [WARNING] e[0m using untested triton version (2.3.0), only 1.0.0 is known to be compatible
e[93m [WARNING] e[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
e[93m [WARNING] e[0m NVIDIA Inference is only supported on Ampere and newer architectures
e[93m [WARNING] e[0m Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
e[93m [WARNING] e[0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
e[93m [WARNING] e[0m async_io: please install the libaio-dev package with apt
e[93m [WARNING] e[0m async_io requires the dev libaio .so object and headers but these were not found.
[2024-06-27 10:31:10,548] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
--dynamo_backend was set to a value of 'no'
The following values were not passed to accelerate launch and had defaults used instead:
INFO | 2024-06-27 10:31:03 | autotrain.backends.local:create:13 - Training PID: 439
INFO | 2024-06-27 10:31:03 | autotrain.commands:launch_command:401 - {âmodelâ: âVatsalPatel18/phi3-mini-WeatherBotâ, âproject_nameâ: âautotrain-uh8dc-qv9mmâ, âdata_pathâ: âtimdettmers/openassistant-guanacoâ, âtrain_splitâ: âtrainâ, âvalid_splitâ: None, âadd_eos_tokenâ: True, âblock_sizeâ: 1024, âmodel_max_lengthâ: 2048, âpaddingâ: ârightâ, âtrainerâ: âsftâ, âuse_flash_attention_2â: False, âlogâ: âtensorboardâ, âdisable_gradient_checkpointingâ: False, âlogging_stepsâ: -1, âeval_strategyâ: âepochâ, âsave_total_limitâ: 1, âauto_find_batch_sizeâ: False, âmixed_precisionâ: âfp16â, âlrâ: 3e-05, âepochsâ: 3, âbatch_sizeâ: 2, âwarmup_ratioâ: 0.1, âgradient_accumulationâ: 4, âoptimizerâ: âadamw_torchâ, âschedulerâ: âlinearâ, âweight_decayâ: 0.0, âmax_grad_normâ: 1.0, âseedâ: 42, âchat_templateâ: ânoneâ, âquantizationâ: âint4â, âtarget_modulesâ: âall-linearâ, âmerge_adapterâ: False, âpeftâ: True, âlora_râ: 16, âlora_alphaâ: 32, âlora_dropoutâ: 0.05, âmodel_refâ: None, âdpo_betaâ: 0.1, âmax_prompt_lengthâ: 128, âmax_completion_lengthâ: None, âprompt_text_columnâ: âpromptâ, âtext_columnâ: âtextâ, ârejected_text_columnâ: ârejected_textâ, âpush_to_hubâ: True, âusernameâ: âhuggingfacepremiumâ, âtokenâ: â*****â, âunslothâ: False}
INFO | 2024-06-27 10:31:03 | autotrain.commands:launch_command:400 - [âaccelerateâ, âlaunchâ, âânum_machinesâ, â1â, âânum_processesâ, â1â, ââmixed_precisionâ, âfp16â, â-mâ, âautotrain.trainers.clmâ, ââtraining_configâ, âautotrain-uh8dc-qv9mm/training_params.jsonâ]
INFO | 2024-06-27 10:31:03 | autotrain.backends.local:create:8 - Starting local trainingâŚ
INFO | 2024-06-27 10:31:03 | autotrain.app.ui_routes:handle_form:491 - hardware: local-ui
INFO | 2024-06-27 10:30:54 | autotrain.app.ui_routes:handle_form:491 - hardware: local-ui
INFO | 2024-06-27 10:26:12 | autotrain.app.utils:kill_process_by_pid:52 - Sent SIGTERM to process with PID 69
INFO | 2024-06-27 10:26:12 | autotrain.app.utils:get_running_jobs:26 - Killing PID: 69
ERROR | 2024-06-27 10:26:08 | autotrain.trainers.common:wrapper:121 - Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
ValueError: Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.
raise ValueError(
File â/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.pyâ, line 627, in _prepare_packed_dataloader
return self._prepare_packed_dataloader(
File â/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.pyâ, line 519, in _prepare_dataset
train_dataset = self._prepare_dataset(
File â/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.pyâ, line 362, in init
return f(*args, **kwargs)
File â/app/env/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.pyâ, line 101, in inner_f
trainer = SFTTrainer(
File â/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/train_clm_sft.pyâ, line 44, in train
train_sft(config)
File â/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/main.pyâ, line 28, in train
return func(*args, **kwargs)
File â/app/env/lib/python3.10/site-packages/autotrain/trainers/common.pyâ, line 117, in wrapper
Traceback (most recent call last):
The above exception was the direct cause of the following exception:
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
raise DatasetGenerationError(âAn error occurred while generating the datasetâ) from e
File â/app/env/lib/python3.10/site-packages/datasets/builder.pyâ, line 1784, in _prepare_split_single
for job_id, done, content in self._prepare_split_single(
File â/app/env/lib/python3.10/site-packages/datasets/builder.pyâ, line 1627, in _prepare_split
self._prepare_split(split_generator, **prepare_split_kwargs)
File â/app/env/lib/python3.10/site-packages/datasets/builder.pyâ, line 1122, in _download_and_prepare
super()._download_and_prepare(
File â/app/env/lib/python3.10/site-packages/datasets/builder.pyâ, line 1789, in _download_and_prepare
self._download_and_prepare(
File â/app/env/lib/python3.10/site-packages/datasets/builder.pyâ, line 1027, in download_and_prepare
self.builder.download_and_prepare(
File â/app/env/lib/python3.10/site-packages/datasets/io/generator.pyâ, line 47, in read
).read()
File â/app/env/lib/python3.10/site-packages/datasets/arrow_dataset.pyâ, line 1125, in from_generator
packed_dataset = Dataset.from_generator(
File â/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.pyâ, line 623, in _prepare_packed_dataloader
Traceback (most recent call last):
The above exception was the direct cause of the following exception:
KeyError: âtextâ
self.formatting_func = lambda x: x[dataset_text_field]
File â/app/env/lib/python3.10/site-packages/trl/trainer/utils.pyâ, line 480, in
buffer.append(self.formatting_func(next(iterator)))
File â/app/env/lib/python3.10/site-packages/trl/trainer/utils.pyâ, line 503, in iter
yield from constant_length_iterator
File â/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.pyâ, line 620, in data_generator
for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
File â/app/env/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.pyâ, line 30, in _generate_examples
for key, record in generator:
File â/app/env/lib/python3.10/site-packages/datasets/builder.pyâ, line 1748, in _prepare_split_single
ERROR | 2024-06-27 10:26:08 | autotrain.trainers.common:wrapper:120 - train has failed due to an exception: Traceback (most recent call last):
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 0 examples [00:00, ? examples/s]
warnings.warn(
/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:307: UserWarning: You passed a dataset_text_field argument to the SFTTrainer, the value you passed will override the one in the SFTConfig.
warnings.warn(
/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:269: UserWarning: You passed a max_seq_length argument to the SFTTrainer, the value you passed will override the one in the SFTConfig.
warnings.warn(
/app/env/lib/python3.10/site-packages/transformers/training_args.py:1965: FutureWarning: --push_to_hub_token is deprecated and will be removed in version 5 of Transformers. Use --hub_token instead.
warnings.warn(
/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:181: UserWarning: You passed a packing argument to the SFTTrainer, the value you passed will override the one in the SFTConfig.
warnings.warn(
/app/env/lib/python3.10/site-packages/transformers/training_args.py:1965: FutureWarning: --push_to_hub_token is deprecated and will be removed in version 5 of Transformers. Use --hub_token instead.
warnings.warn(message, FutureWarning)
Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead.
/app/env/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in âinitâ: dataset_text_field, max_seq_length, packing. Will not be supported from version â1.0.0â.
INFO | 2024-06-27 10:26:07 | autotrain.trainers.clm.train_clm_sft:train:37 - creating trainer
INFO | 2024-06-27 10:26:07 | autotrain.trainers.clm.utils:get_model:666 - model dtype: torch.float16
Loading checkpoint shards: 100%|ââââââââââ| 2/2 [00:13<00:00, 6.74s/it]
Loading checkpoint shards: 100%|ââââââââââ| 2/2 [00:13<00:00, 6.25s/it]
Downloading shards: 100%|ââââââââââ| 2/2 [00:36<00:00, 18.12s/it]
Downloading shards: 100%|ââââââââââ| 2/2 [00:36<00:00, 17.92s/it]
low_cpu_mem_usage was None, now set to True since model is quantized.
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
- modeling_phi3.py
A new version of the following files was downloaded from https://huggingface.co/microsoft/Phi-3-mini-4k-instruct:
INFO | 2024-06-27 10:25:17 | autotrain.trainers.clm.utils:get_model:635 - loading modelâŚ
INFO | 2024-06-27 10:25:17 | autotrain.trainers.clm.utils:get_model:627 - loading model configâŚ
WARNING | 2024-06-27 10:25:17 | autotrain.trainers.clm.utils:get_model:625 - Unsloth not available, continuing without itâŚ
INFO | 2024-06-27 10:25:17 | autotrain.trainers.clm.utils:get_model:583 - Can use unsloth: False
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
- configuration_phi3.py
A new version of the following files was downloaded from huggingface.co/microsoft/Phi-3-mini-4k-instruct:
warnings.warn(
/app/env/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
INFO | 2024-06-27 10:25:16 | autotrain.trainers.clm.utils:configure_block_size:548 - Using block size 1024
INFO | 2024-06-27 10:25:16 | autotrain.trainers.clm.utils:configure_training_args:485 - configuring training args
INFO | 2024-06-27 10:25:16 | autotrain.trainers.clm.utils:configure_logging_steps:480 - Logging steps: 25
INFO | 2024-06-27 10:25:16 | autotrain.trainers.clm.utils:configure_logging_steps:467 - configuring logging steps
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO | 2024-06-27 10:25:16 | autotrain.trainers.clm.utils:process_input_data:395 - Valid data: None
})
num_rows: 16636
features: [âquestionâ, âanswerâ],
INFO | 2024-06-27 10:25:16 | autotrain.trainers.clm.utils:process_input_data:394 - Train data: Dataset({
Generating train split: 100%|ââââââââââ| 16636/16636 [00:00<00:00, 112909.06 examples/s]
Generating train split: 100%|ââââââââââ| 16636/16636 [00:00<00:00, 115699.95 examples/s]
Generating train split: 0%| | 0/16636 [00:00<?, ? examples/s]
Downloading data: 100%|ââââââââââ| 9.00M/9.00M [00:00<00:00, 19.1MB/s]
Downloading data: 100%|ââââââââââ| 9.00M/9.00M [00:00<00:00, 19.2MB/s]
Downloading data: 0%| | 0.00/9.00M [00:00<?, ?B/s]
Downloading readme: 100%|ââââââââââ| 145/145 [00:00<00:00, 1.12MB/s]
Downloading readme: 0%| | 0.00/145 [00:00<?, ?B/s]
INFO | 2024-06-27 10:25:15 | autotrain.trainers.clm.train_clm_sft:train:12 - Starting SFT trainingâŚ
e[93m [WARNING] e[0m using untested triton version (2.3.0), only 1.0.0 is known to be compatible
e[93m [WARNING] e[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
e[93m [WARNING] e[0m NVIDIA Inference is only supported on Ampere and newer architectures
e[93m [WARNING] e[0m Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
e[93m [WARNING] e[0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
e[93m [WARNING] e[0m async_io: please install the libaio-dev package with apt
e[93m [WARNING] e[0m async_io requires the dev libaio .so object and headers but these were not found.
[2024-06-27 10:25:14,784] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
--dynamo_backend was set to a value of 'no'
The following values were not passed to accelerate launch and had defaults used instead:
INFO | 2024-06-27 10:25:07 | autotrain.backends.local:create:13 - Training PID: 69
INFO | 2024-06-27 10:25:07 | autotrain.commands:launch_command:401 - {âmodelâ: âmicrosoft/Phi-3-mini-4k-instructâ, âproject_nameâ: âautotrain-uh8dc-qv9uaâ, âdata_pathâ: âhuggingfacepremium/trainâ, âtrain_splitâ: âtrainâ, âvalid_splitâ: None, âadd_eos_tokenâ: True, âblock_sizeâ: 1024, âmodel_max_lengthâ: 2048, âpaddingâ: ârightâ, âtrainerâ: âsftâ, âuse_flash_attention_2â: False, âlogâ: âtensorboardâ, âdisable_gradient_checkpointingâ: False, âlogging_stepsâ: -1, âeval_strategyâ: âepochâ, âsave_total_limitâ: 1, âauto_find_batch_sizeâ: False, âmixed_precisionâ: âfp16â, âlrâ: 3e-05, âepochsâ: 3, âbatch_sizeâ: 2, âwarmup_ratioâ: 0.1, âgradient_accumulationâ: 4, âoptimizerâ: âadamw_torchâ, âschedulerâ: âlinearâ, âweight_decayâ: 0.0, âmax_grad_normâ: 1.0, âseedâ: 42, âchat_templateâ: ânoneâ, âquantizationâ: âint4â, âtarget_modulesâ: âall-linearâ, âmerge_adapterâ: False, âpeftâ: True, âlora_râ: 16, âlora_alphaâ: 32, âlora_dropoutâ: 0.05, âmodel_refâ: None, âdpo_betaâ: 0.1, âmax_prompt_lengthâ: 128, âmax_completion_lengthâ: None, âprompt_text_columnâ: âpromptâ, âtext_columnâ: âtextâ, ârejected_text_columnâ: ârejected_textâ, âpush_to_hubâ: True, âusernameâ: âhuggingfacepremiumâ, âtokenâ: â*****â, âunslothâ: False}
INFO | 2024-06-27 10:25:07 | autotrain.commands:launch_command:400 - [âaccelerateâ, âlaunchâ, âânum_machinesâ, â1â, âânum_processesâ, â1â, ââmixed_precisionâ, âfp16â, â-mâ, âautotrain.trainers.clmâ, ââtraining_configâ, âautotrain-uh8dc-qv9ua/training_params.jsonâ]
INFO | 2024-06-27 10:25:07 | autotrain.backends.local:create:8 - Starting local trainingâŚ
INFO | 2024-06-27 10:25:07 | autotrain.app.ui_routes:handle_form:491 - hardware: local-ui
Blockquote