Fine-tuning Whisper for Translation

Beginners
1

Hi @sanchit-gandhi @MariaK, I have been following the guide to fine-tune whisper in the course and also looking at this blog. Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers

I wanted to know that if I want to fine-tune for translation instead of transcription, I only need to change the task in the WhisperAudioProcessor and the metrics (using BLEU instead of WER) in the compute metrics function right?

Do I need to make any other changes to modify the code for translation to English?

Best,
Raj

1 Like
2

@RajSang have you figured it out…what I think is we might also need a paired speech to English text data.

3

up, I have the same question

4

I’ve done a project where I fine-tuned Whisper-Tiny for translation tasks, and it gave good results. You can take a look at my repo to see the steps I used, which might help you solve the issue you’re facing.
MediBeng-Whisper-Tiny Github repo:

1 Like