Performing Whisper's "transcribe" with Transformer pipelines

[panigrah]

For your side quest; Add a torch.set_default_device(device) prior to loading the model. The message is harmless anyway as you are moving the model to a device after creating it.

@T145

You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU

For additional tokens - I believe you have to train, finetune because adding tokens causes the model to lose some of its weights.

Adding custom vocabularies on Whisper

Hey @sebasarango1180, If you have a corpus of paired audio-text data with examples of such terms/entities/acronyms, you could experiment with fine-tuning the Whisper model on this dataset and seeing whether this improves downstream ASR performance on this distribution of data. To do so, you can follow the blog post at Fine-Tune Whisper For Multilingual ASR with Transformers. Since we’re fine-tuning for English-only (rather than multilingual), we need to make two modifications to the recipe o…