Adding prompt / context to Whisper with Huggingface Transformers

Models
1

The Whisper model, has the possibility of a prompt or adding the previous text to the current transcription task. This helps in case of transcribing long file chunk after chunk.

During training it should “mask out the training loss over the previous context text, and train the
model to predict all other tokens”.

I’m wondering if HF has implemented that and how well does it helps with the accuracy.

There is also this issue in Huggingface Transformers:

I’m wondering if it is fixed and stable.
There it seems that the way of doing it with HF would be is something like so:

prev_before = '<|startofprev|>'
current_before = "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>"
current_after = "<|endoftext|>"

prev_text = 'Hello and'
current_text = 'welcome. My name is...'

prompt_and_text = prev_before + prev_text + current_before + current_text + current_after
prompt_and_text

<|startofprev|>Hello and<|startoftranscript|><|en|><|transcribe|><|notimestamps|>welcome. My name is…<|endoftext|>

prompt_and_text_tokens = _tokenizer.encode(prompt_and_text, add_special_tokens=False)
_tokenizer.decode(prompt_and_text_tokens)

<|startofprev|>Hello and<|startoftranscript|><|en|><|transcribe|><|notimestamps|>welcome. My name is…<|endoftext|>

2 Likes
2

@SamuelAzran did you manage to get this working? I’d like to have the exact same functionality.

3

Any news on this?

4

I did not see Huggingface supporting it or providing an easy way to do it, like in the open_ai library, but you can hack around or use lower-level components in HF to give context or prompt in inference time. In training time, training on prompts is much more complicated. I hope that someone from HF can address this issue.

The benefits of using prompts are beyond just giving the text of the previous speech segments. You can also provide prompts to control the style of the transcript and add relevant domain-specific or industry-specific terms to the context so the transcription process will be more likely to use those terms in the final transcript.

1 Like
5

Great! Thanks! That’s what I will end up doing

6

Related discussion: Is it possible to do model adaptation? · openai/whisper · Discussion #66 · GitHub

You can also use logit biasing to increase the probability of certain domain specific words being predicted.

7

Any update on support for this in hf libraries?

8

For anyone who is still looking for a solution.
I think I manged to make it work. Here is an example with whisper medium in French.

First extract the prompt ids with get_prompt_ids() method:

import torch
prev_text = "est ce qu'ils sont choqués ?"
prompt_ids = torch.tensor(whisper_processor.get_prompt_ids(prev_text), device=device)
print(whisper_processor.decode(prompt_ids, skip_special_tokens=False))

“<|startofprev|> est ce qu’ils sont choqués ?”

Then inject this prompt in inference. this inference uses no sampling to remain deterministic in order to show the real effect of the prompt:

audio_features = extract_audio_input_features(
    audio,
    whisper_processor,
    device,
    torch_dtype
)
whisper_model.generation_config.forced_decoder_ids = None
predicted_ids = whisper_model.generate(
    audio_features,
    language="fr",
    task="transcribe",
    eos_token_id=whisper_model.generation_config.eos_token_id,
    pad_token_id=whisper_model.generation_config.pad_token_id,
    max_time=5,
    do_sample=False,
    return_dict_in_generate=False,
    prompt_ids=prompt_ids
)

print(whisper_processor.decode(predicted_ids[0]))

“Non, ça va, pour l’instant ça va, ils sont choqués mais ça va.”

Here is the original transcription, without the prompt (also without any random sampling):
“Non, ça va. Pour l’instant, ça va. Ils sont pour chocrer, mais ça va.”
(Error on word “choqués”)

I’m open to remarks or ways to improve this.

1 Like