For now, after looking into this, it seems quite likely that the issue is related to a bug on the FunASR side:
I do not think your intended setup is fundamentally unsupported.
The current funasr/paraformer-zh Hugging Face model card explicitly shows the full pipeline with VAD, punctuation, and speaker diarization, and says that the output includes timestamps and speaker labels. So I would not abandon the HF route or the diarization goal just because of this error.
What I found is closer to two separate timestamp-related problems in FunASR, plus a secondary output-schema issue.
The cheapest first test is simply:
result = model.generate(
input=audio,
pred_timestamp=True,
)
In controlled reproductions with the HF Paraformer model, this was enough to change the ASR result from no timestamp to timestamp present, on both CPU and T4.
So, for practical debugging, I would start there before changing the model stack.
A rough decision flow is:
add pred_timestamp=True
|
+-- still no timestamp
| -> check which model/config actually resolved
|
+-- timestamp appears
|
+-- timestamps stay within the audio duration
| -> continue checking the speaker pipeline
|
+-- timestamps extend well beyond the audio duration
-> likely the separate GPU dynamic-batching issue below
There is also a public reproduction notebook here:
1. Why the missing timestamp looks like an internal FunASR integration issue
The interesting part is that the composite pipeline and the underlying Paraformer implementation currently appear to use different timestamp-request names.
In current AutoModel, when a speaker model is present, the VAD pipeline enables:
cfg["output_timestamp"] = True
cfg["return_time_stamps"] = True
But the current plain Paraformer.inference() checks:
pred_timestamp = kwargs.get("pred_timestamp", False)
and only enters its timestamp-generation branch when that value is true.
So the effective path looks roughly like this:
spk_model is enabled
|
v
AutoModel requests timestamp output
output_timestamp=True
return_time_stamps=True
|
v
plain Paraformer inference
|
+-- checks pred_timestamp
|
+-- pred_timestamp is still False
|
v
no ASR timestamp
That matches the runtime A/B quite closely:
HF Paraformer baseline
-> no timestamp
same loaded model
+ pred_timestamp=True
-> timestamp appears
The same result was reproduced on CPU as well as T4, so this particular missing-timestamp problem does not look CUDA-specific.
This is why I currently suspect a wrapper/model timestamp-contract mismatch rather than a basic problem with your use of Hugging Face Hub.
There is also some historical precedent for timestamp availability causing composite FunASR pipelines to fail. For example, an older FunASR issue reported a KeyError: 'timestamp' when combining Paraformer, VAD and speaker processing:
FunASR issue #1747 — Paraformer + VAD result missing timestamp
I would treat that as related evidence rather than proof that the exact same code path is involved.
There is also a similar integration theme in:
FunASR issue #2706 — SenseVoice with CAM++ speaker diarization
Again, not necessarily the same root cause, but it is useful precedent that speaker processing can depend on model-specific timestamp activation.
2. The `KeyError: 'text'` looks secondary rather than the root problem
I would be careful about treating the final:
KeyError: 'text'
as the first failure.
In the controlled HF pipeline reproduction, the output schema changed depending on whether the normal timestamp path was available.
Without the timestamp path, the first sentence_info item used a field named:
"sentence"
rather than:
"text"
With pred_timestamp=True, the normal richer path returned the documented:
"text"
field again.
That matters because the current HF model card shows:
for sentence in result[0]["sentence_info"]:
print(f"[Speaker {sentence['spk']}] {sentence['text']}")
So the observed sequence can be:
timestamp was not generated
|
v
pipeline takes a fallback path
|
v
sentence_info has a different schema
|
v
code expects sentence["text"]
|
v
KeyError: "text"
In other words, changing every text access to sentence may hide the last exception, but I would not regard that as the primary fix.
A cheap diagnostic is just:
print(result[0].keys())
if result[0].get("sentence_info"):
print(result[0]["sentence_info"][0].keys())
If you need defensive application code while testing, you can of course tolerate both field names, but I would still fix/understand the timestamp path first.
3. There appears to be a second, independent GPU dynamic-batching timestamp bug
After explicitly enabling pred_timestamp=True, I hit a second issue that is independent of the missing-timestamp problem.
A pinned 20.000 s AISHELL-4 crop behaved approximately like this:
| execution path |
maximum timestamp end |
| physical audio |
20.000 s |
| whole crop / sequential references |
~20.03–20.04 s |
| GPU forced one-VAD-segment batches |
20.030 s |
| ordinary GPU dynamic VAD batch |
24.710 s |
So the large +4.71 s error was not a general timestamp offset.
It appeared specifically when unequal-length VAD segments were grouped into one GPU batch.
The useful internal trace was:
encoder_out_lens:
[64, 73, 142]
timestamp predictor tensor width:
143
Before any fix, the timestamp helper received:
sample 0: 143
sample 1: 143
sample 2: 143
even though the first two samples had much shorter real encoder lengths.
For the tested HF Mandarin model, the runtime predictor was CifPredictorV2 with:
tail_mask=True
tail_threshold=0.45
The valid timestamp extent for that branch is therefore the sample encoder length plus the one CIF tail frame:
[64 + 1, 73 + 1, 142 + 1]
=
[65, 74, 143]
For the shortest sample:
143 - 65 = 78 extra frames
Plain Paraformer calls the timestamp helper with upsample_rate=1, which corresponds to about 60 ms per frame here:
78 * 60 ms = 4680 ms
The actual abnormal tail was:
24.710 s - 20.000 s = 4710 ms
That numerical agreement is unusually close.
More importantly, this was tested causally rather than only inferred from the numbers.
I left the normal GPU dynamic batching unchanged and changed only the tensors passed into timestamp conversion, trimming each sample to its valid extent.
The result was:
unmodified dynamic batch
-> 24.710 s
-> 1 backward timestamp jump
same dynamic batch
timestamp input trimmed per sample
-> 20.030 s
-> 0 backward jumps
forced-single reference
-> 20.030 s
-> 0 backward jumps
The recognized text before and after the timestamp-only trim was identical, and the timestamp count remained 67 → 67.
The timestamp-helper widths changed from:
[143, 143, 143]
to:
[65, 74, 143]
So for the tested Mandarin Paraformer path, I think there is fairly strong evidence that the large timestamp tail comes from batch-max padded tensor extent leaking into timestamp conversion.
The current source still has the relevant shape.
Plain Paraformer calls:
ts_prediction_lfr6_standard(
pre_peak_index[i],
alphas[i],
...
)
without passing/slicing by that sample’s encoder_out_lens[i]:
Paraformer timestamp path
And the shared helper derives its end extent from the tensor it receives:
num_frames = peaks.shape[0]
See:
timestamp_tools.py
There is also useful precedent inside FunASR itself. BiCifParaformer slices the timestamp tensors using the per-sample encoder length before calling the same helper:
us_alphas[i][: encoder_out_lens[i] * 3]
us_peaks[i][: encoder_out_lens[i] * 3]
See:
bicif_paraformer/model.py
So per-sample timestamp extent is not a foreign idea to the existing codebase.
4. Practical workaround versus the narrow fix
For actually getting your pipeline moving, I would separate the workaround from the possible upstream fix.
Low-cost route
First try:
res = model.generate(
input="meeting.wav",
pred_timestamp=True,
)
If the timestamps now appear and remain sensible, that may be enough for your immediate case.
If GPU timestamps overshoot the audio duration
As a temporary diagnostic/workaround, forcing the VAD segments out of ordinary dynamic grouping removed the large timestamp tail in my reproduction.
For example, the tested path used:
res = model.generate(
input="meeting.wav",
pred_timestamp=True,
batch_size_threshold_s=0,
)
That should not be interpreted as the ideal production solution: it can change batching/performance behavior.
But it is a very useful discriminator:
ordinary dynamic batching -> bad timestamps
forced-single batching -> normal timestamps
If you see that same split, it strongly points toward the batching/timestamp-length issue rather than VAD boundaries or general CIF quality.
Narrow implementation-side fix
For the validated CifPredictorV2(tail_mask=True) path, the minimal idea is:
timestamp_pre_peak_index = pre_peak_index[i]
timestamp_alphas = alphas[i]
if getattr(self.predictor, "tail_mask", None) is True:
timestamp_len = int(encoder_out_lens[i].item())
if float(getattr(self.predictor, "tail_threshold", 0.0)) > 0.0:
timestamp_len += 1
timestamp_len = min(
timestamp_len,
timestamp_pre_peak_index.shape[-1],
timestamp_alphas.shape[-1],
)
timestamp_pre_peak_index = timestamp_pre_peak_index[:timestamp_len]
timestamp_alphas = timestamp_alphas[:timestamp_len]
and then pass those tensors to ts_prediction_lfr6_standard().
The conservative guard is intentional.
It does not alter:
tail_mask=False
and it does not assume that predictors without a tail_mask attribute have identical semantics.
The full exact diff, CPU-only deterministic regression, and real-model T4 reproduction are in the public notebooks linked above.
The executed run verifies all of the following together:
20.000 s input
unmodified dynamic:
max end = 24.710 s
patched dynamic:
max end = 20.030 s
patched forced-single:
max end = 20.030 s
recognized dynamic text:
unchanged
timestamp count:
67 -> 67
helper widths:
[143,143,143] -> [65,74,143]
original installed source:
restored after the experiment
For an upstream regression test, this can also be checked without downloading the real model: a dummy Paraformer.inference() test with unequal encoder lengths can verify that the timestamp helper sees the per-sample extent. That makes a fairly small, deterministic CPU regression possible.
5. `hub='hf'` versus `hub='ms'` is worth interpreting carefully
One other thing that can make this confusing: changing:
hub="hf"
to:
hub="ms"
is not necessarily a pure test of:
same checkpoint + same architecture + different download host
when short aliases such as:
model="paraformer-zh"
are used.
In the environment I checked, the HF alias resolved to the plain HF Paraformer model, while the ModelScope-side alias could resolve through a SeACoParaformer model path.
There is an existing FunASR report from someone surprised that paraformer-zh caused the ModelScope SeACoParaformer model to be downloaded:
FunASR issue #2501
That does not prove that the mapping is wrong; it may be intentional compatibility/alias behavior.
It does mean that:
HF works differently from ModelScope
does not automatically imply:
Hugging Face downloaded the same model incorrectly
It may actually be a model-family comparison.
A cheap way to remove that ambiguity is:
print(type(model.model).__name__)
print(type(model.model).__module__)
print(model.model_path)
Using fully qualified model IDs where possible also makes the comparison easier to interpret.
For reference, the currently documented HF pipeline uses:
model = AutoModel(
model="funasr/paraformer-zh",
hub="hf",
vad_model="funasr/fsmn-vad",
punc_model="funasr/ct-punc",
spk_model="funasr/campplus",
device="cuda",
)
from the funasr/paraformer-zh model card.
6. What I would not conclude from these results
A few boundaries seem important here.
I would not say that FunASR as a whole is broken
The evidence is much narrower:
- HF plain Paraformer timestamp activation in the composite pipeline;
- fallback schema behavior after timestamp absence;
- per-sample timestamp extent under GPU dynamic batching.
I would not say that Hugging Face Hub itself is the root cause
The HF model loads and runs. The strongest evidence points inside FunASR’s model/wrapper/timestamp processing after loading.
I would not treat speaker-path execution as proof that diarization accuracy is fixed
Restoring timestamps lets the intended speaker pipeline operate, but:
pipeline runs
and:
speaker assignments are accurate
are different questions.
I did not measure DER or otherwise validate diarization quality here.
I would not generalize this timestamp trim to every Paraformer-family model yet
The runtime fix was validated on the HF Mandarin paraformer-zh path with:
CifPredictorV2
tail_mask=True
tail_threshold=0.45
tail_mask=False, legacy predictor implementations, EParaformer, etc. should be treated separately until they have their own runtime checks.
I would also avoid globally translating every timestamp flag into pred_timestamp=True for every model
That sounds attractive as a generic API cleanup, but broader testing exposed a separate English/BPE timestamp-path problem.
The FunASR repository itself currently describes paraformer-zh as a model with timestamps, while paraformer-en is listed as a model without timestamps:
FunASR repository/model list
So I would keep the Mandarin HF fix narrow rather than making a global timestamp-policy change at the same time.
So my default route would be:
- Keep your intended HF Paraformer + VAD + punctuation + speaker setup.
- Add
pred_timestamp=True first.
- Inspect whether timestamps now exist.
- If they exist but extend far beyond the audio duration on GPU, compare ordinary dynamic batching with a forced-single reference.
- Treat a
KeyError: 'text' after missing timestamps as likely secondary, and inspect the actual sentence_info keys rather than assuming the documented schema was reached.
- If you need an implementation-level fix for the dynamic-batch case, the per-sample timestamp trim above is the narrowest version I have been able to validate so far.
So, at least from these reproductions, this looks much more like a couple of FunASR integration/length-handling bugs that can be separated and worked around than a reason to give up on what you are trying to build.