How to fine-tune whisper on unsupported language?

andrewbawitlung · October 11, 2024, 10:19am

While following this blog,

What should I use for these lines while fine-tuning on another language which is not supported?

model.generation_config.language = "hindi"

John6666 · October 12, 2024, 4:44am

There definitely seems to be a way. I’ve never tried to adjust a model, so I wouldn’t count on it…

github.com/huggingface/transformers

How to set language in Whisper pipeline for audio transcription?

opened 05:41PM - 26 Feb 23 UTC

closed 01:37PM - 27 Feb 23 UTC

melihogutcen

### Problem Hello, I followed this notebook for Whisper pipelines. https:…//colab.research.google.com/drive/1rS1L4YSJqKUH_3YxIQHBI982zso23wor?usp=sharing#scrollTo=Ca4YYdtATxzo Here, I want to use speech transcription with openai/whisper-large-v2 model using the pipeline. By using WhisperProcessor, we can set the language, but this has a disadvantage for longer audio files than 30 seconds. I used the below code and I can set the language here. ``` import torch from transformers import WhisperForConditionalGeneration, WhisperProcessor device = "cuda" if torch.cuda.is_available() else "cpu" model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2").to(device) processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2") inputs = processor.feature_extractor(speech_data, return_tensors="pt", sampling_rate=16_000).input_features.to(device) generate_ids = model.generate(inputs, max_length=480_000, language="<|tr|>", task="transcribe", return_timestamps=True) results = processor.tokenizer.decode(generate_ids[0], decode_with_timestamps=True, output_offsets=True) ``` Long audio files can be processed in the pipeline by setting chunk_length as below. But in the pipeline, I couldn't set the language. Therefore, I have gotten English results in my Turkish speech data. ``` from transformers import pipeline MODEL_NAME = "openai/whisper-large-v2" pipe = pipeline( task="automatic-speech-recognition", model=MODEL_NAME, device='cpu') pipe(speech_file, return_timestamps=True, chunk_length_s=30, stride_length_s=[6,0], batch_size=32) ``` Is there a way to set the language? ### System Info docker image: - pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime Transformers Version: `transformers==v4.27dev` ### Who can help? @sanchit-gandhi @Narsil ### Information - [X] The official example scripts - [ ] My own modified scripts ### Tasks - [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ### Reproduction ``` from transformers import pipeline MODEL_NAME = "openai/whisper-large-v2" pipe = pipeline( task="automatic-speech-recognition", model=MODEL_NAME, device='cpu') pipe(speech_file, return_timestamps=True, chunk_length_s=30, stride_length_s=[6,0], batch_size=32) ``` ### Expected behavior ``` Label: "Bazı Türkçe kelimeler." Prediction: "Some Turkish words." ```

Topic		Replies	Views
How to use Whisper from huggingface for ASR DeepSpeed	0	515	June 21, 2023
Open ai whisper fine tuning on unknown language Beginners	0	40	October 1, 2024
Replicating the Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers on MPS Beginners	0	148	July 24, 2024
How to set language in Whisper pipeline for audio transcription? 🤗Transformers	2	6404	June 22, 2023
Performing Whisper's "transcribe" with Transformer pipelines Beginners	2	2016	December 19, 2023

How to fine-tune whisper on unsupported language?

Related topics