I’m trying to fine-tune Whisper for French. I used this code to get started: Fine-Tune Whisper For Multilingual ASR with 🤗 Transformers. I’ve run a few tests with different parameters like learning rate, batch size, max steps, and warmup ratio. I’m using the French Common Voice dataset.
The training resulted in better performance on WER and stopped prompting on silence, but the model no longer recognizes some words it knew before. I’d like to find a solution for this issue. This dataset won’t be my final one, as I plan to add vocabulary next, but I’m worried about losing everything Whisper has already learned. And loosing my next fine-tuning too.
Thanks for your help! 