Hm. It seems to have improved quite a bit. From here, I’d probably think about it like this:
The MFCC/log-Mel + SVM result is the key new information.
If the same grouped 5-fold setup gives around:
Accuracy: 90.7% ± 2.1%
Macro-F1: 89.4% ± 2.2%
Majority baseline: 50.5%
then I would now treat the classes as acoustically separable. So the remaining question is probably not:
Is there any signal in the labels?
but rather:
Why is the Wav2Vec2/WavLM sequence classifier not preserving or exposing the same local acoustic information that the MFCC/log-Mel baseline can use?
So yes, I think your changes helped. But from here, I would focus less on the backbone name and more on the pooling / aggregation head.
Direct answer: how the default pooling works
In Wav2Vec2ForSequenceClassification / WavLMForSequenceClassification, the rough path is:
raw waveform
→ SSL encoder
→ hidden states: [batch, time, hidden]
→ optional weighted layer sum
→ projector
→ time pooling
→ classifier
The important part is the time pooling. In the Transformers implementation, after projection, the default sequence-classification head does roughly this:
if attention_mask is None:
pooled_output = hidden_states.mean(dim=1)
else:
padding_mask = self._get_feature_vector_attention_mask(
hidden_states.shape[1],
attention_mask,
)
hidden_states[~padding_mask] = 0.0
pooled_output = hidden_states.sum(dim=1) / padding_mask.sum(dim=1).view(-1, 1)
So the default classifier is basically using mean pooling over time, or masked mean pooling if an attention mask is passed.
Relevant source files:
That default is a reasonable generic baseline, but your task is not a broad utterance-level classification task. It is a short sibilant/fricative articulation classification task. A plain average over time may be too lossy.
My short recommendation
I would try pooling heads in this order:
| priority |
pooling/head |
why |
| 1 |
masked mean + std pooling |
smallest change from default mean pooling; preserves temporal variation |
| 2 |
central / segment-frame pooling |
focuses on the target fricative instead of all frames |
| 3 |
attention pooling |
lets the model learn which frames matter |
| 4 |
small TDNN/CNN head + statistic pooling |
gives the head local temporal pattern capacity |
| 5 |
weighted layer sum + better pooling |
tests whether another layer contains better phonetic cues |
If I had to pick only one first experiment, I would try:
same setup as your current Wav2Vec2/WavLM run
but replace default masked mean pooling
with masked mean + std pooling
Mean pooling asks:
What is the average hidden representation?
Mean + std pooling asks:
What is the average hidden representation, and how does it vary over time?
For a fricative/sibilant task, that second part may matter a lot.
Why mean pooling may be too weak here
Your acoustic baseline result is important because MFCC/log-Mel features preserve fairly direct local spectral information.
The Wav2Vec2/WavLM sequence-classification head, however, compresses the hidden sequence into one vector. With default pooling, local frame-level differences can be smoothed away.
For example, the model may see hidden states shaped like:
[batch, time, hidden]
Then default pooling turns that into:
[batch, hidden]
by averaging over time.
That can work well for tasks where the whole utterance carries the label. But for your case, the discriminative cue may be something like:
- spectral concentration,
- turbulence/noise shape,
- lateralized energy distribution,
- short local cue in the fricative,
- difference between the central part and the edges of the segment.
A single mean vector may not preserve enough of that.
The fact that MFCC/log-Mel + SVM gets around 90% suggests that the signal is real and relatively accessible in acoustic-feature space. The Wav2Vec2/WavLM issue now looks more like:
good signal exists
but the SSL sequence classifier is not aggregating it well
rather than:
the labels are not learnable
Pooling option 1: masked mean + std pooling
This is the first thing I would try.
Instead of using only:
mean = hidden_states.mean(dim=1)
use:
pooled = concat(mean, std)
Very rough implementation:
def masked_mean_std_pooling(hidden_states, padding_mask=None, eps=1e-8):
# hidden_states: [B, T, H]
# padding_mask: [B, T], True for valid frames
if padding_mask is None:
mean = hidden_states.mean(dim=1)
std = hidden_states.std(dim=1)
return torch.cat([mean, std], dim=-1)
mask = padding_mask.unsqueeze(-1).to(hidden_states.dtype) # [B, T, 1]
lengths = mask.sum(dim=1).clamp(min=1.0) # [B, 1]
mean = (hidden_states * mask).sum(dim=1) / lengths
var = ((hidden_states - mean.unsqueeze(1)) ** 2 * mask).sum(dim=1) / lengths
std = torch.sqrt(var + eps)
return torch.cat([mean, std], dim=-1)
Then the classifier input size becomes double:
projected_hidden_size → 2 * projected_hidden_size
Why this is a good first test:
| reason |
explanation |
| close to default |
easy ablation against current pooling |
| still simple |
no extra attention module yet |
| preserves variation |
useful if articulation differs in local temporal/spectral shape |
| known pattern |
statistic pooling is common in speaker-style heads |
There is a related precedent inside Transformers itself: the Wav2Vec2/WavLM XVector models use TDNN layers followed by statistic pooling, i.e. mean + std pooling.
Again, this does not mean XVector is the right model for your task. But it does make statistic pooling a natural first alternative to plain mean pooling.
Relevant source areas:
Pooling option 2: central / segment-frame pooling
Since the clips are already centered around a target phone, another useful test is to pool only the central part of the hidden sequence.
Crude example:
def central_pooling(hidden_states, center_fraction=0.5):
# hidden_states: [B, T, H]
B, T, H = hidden_states.shape
width = max(1, int(T * center_fraction))
start = max(0, (T - width) // 2)
end = start + width
return hidden_states[:, start:end].mean(dim=1)
You can also combine this with std:
central mean + central std
This tests a simple hypothesis:
Is the model failing because it averages over too much non-target context?
Even with the new 0.3s padding limit, your median clip is ~0.16s, so the exact pooled region may still matter.
Possible variants:
| variant |
what it tests |
| full masked mean |
generic utterance-level summary |
| central 50% mean |
more target-focused summary |
| central 50% mean + std |
target-focused statistic pooling |
| central fixed N frames |
stable phone-centered representation |
| max over central frames |
preserve strong local activations |
This is not elegant, but it is a good diagnostic.
Pooling option 3: attention pooling
Attention pooling lets the model learn which frames matter.
Minimal example:
class AttentionPooling(torch.nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.score = torch.nn.Linear(hidden_size, 1)
def forward(self, hidden_states, padding_mask=None):
# hidden_states: [B, T, H]
scores = self.score(hidden_states).squeeze(-1) # [B, T]
if padding_mask is not None:
scores = scores.masked_fill(~padding_mask, -1e9)
weights = torch.softmax(scores, dim=-1) # [B, T]
pooled = torch.sum(hidden_states * weights.unsqueeze(-1), dim=1)
return pooled
This can be stronger than mean pooling because it can learn:
these frames matter more than those frames
But I would try it after mean+std, because attention pooling adds more flexibility and can overfit more easily on a small dataset.
Potential diagnostics:
| check |
why |
| inspect attention weights |
see whether it focuses on the target segment |
| compare to central pooling |
distinguish learned focus from simple center bias |
| use dropout |
reduce overfitting |
| keep folds identical |
make the ablation interpretable |
Pooling option 4: small TDNN/CNN head over hidden states
Because MFCC/log-Mel + SVM works well, the discriminative cue may be local and spectral. A small temporal head over SSL hidden states may help more than a pure pooling change.
Conceptually:
hidden states [B, T, H]
→ small temporal CNN / TDNN
→ nonlinearity
→ statistic pooling
→ classifier
Rough shape:
class TemporalConvPoolingHead(torch.nn.Module):
def __init__(self, hidden_size, channels=256, num_labels=3):
super().__init__()
self.conv = torch.nn.Sequential(
torch.nn.Conv1d(hidden_size, channels, kernel_size=3, padding=1),
torch.nn.ReLU(),
torch.nn.Dropout(0.1),
torch.nn.Conv1d(channels, channels, kernel_size=3, padding=1),
torch.nn.ReLU(),
)
self.classifier = torch.nn.Linear(channels * 2, num_labels)
def forward(self, hidden_states, padding_mask=None):
# hidden_states: [B, T, H]
x = hidden_states.transpose(1, 2) # [B, H, T]
x = self.conv(x)
x = x.transpose(1, 2) # [B, T, C]
pooled = masked_mean_std_pooling(x, padding_mask)
logits = self.classifier(pooled)
return logits
This is still much lighter than redesigning the whole model, but it gives the classifier local temporal pattern capacity.
I would only try this after simpler pooling ablations, because it adds more moving parts.
Important implementation detail: feature-vector mask
If you build a custom pooling head, do not directly use the waveform-level attention_mask as if it had the same time dimension as last_hidden_state.
The waveform attention mask is over:
input samples
The hidden state sequence is over:
post-convolution feature frames
So you need a feature-vector-level mask.
The built-in sequence classifier does this internally with:
self._get_feature_vector_attention_mask(
hidden_states.shape[1],
attention_mask,
)
This is the same kind of issue discussed here:
https://github.com/huggingface/transformers/issues/25307
For quick experiments, you can use the model’s helper method if available. Just be aware that internal/private helper methods can be version-sensitive.
Pseudo-shape:
outputs = model.wav2vec2(
input_values=input_values,
attention_mask=attention_mask,
output_hidden_states=True,
return_dict=True,
)
hidden_states = outputs.last_hidden_state # [B, T_feat, H]
padding_mask = model._get_feature_vector_attention_mask(
hidden_states.shape[1],
attention_mask,
)
Then use padding_mask for your custom pooling.
Small custom head sketch
This is only a rough sketch, not drop-in final code. The point is the architecture:
SSL hidden sequence
→ projector
→ masked mean + std pooling
→ classifier
import torch
import torch.nn as nn
from transformers import AutoModel, AutoConfig
class SSLMeanStdClassifier(nn.Module):
def __init__(self, model_name, num_labels, classifier_hidden=256):
super().__init__()
self.config = AutoConfig.from_pretrained(model_name)
self.backbone = AutoModel.from_pretrained(model_name)
hidden_size = self.config.hidden_size
self.projector = nn.Linear(hidden_size, classifier_hidden)
self.classifier = nn.Linear(classifier_hidden * 2, num_labels)
def _masked_mean_std(self, hidden_states, padding_mask=None, eps=1e-8):
if padding_mask is None:
mean = hidden_states.mean(dim=1)
std = hidden_states.std(dim=1)
return torch.cat([mean, std], dim=-1)
mask = padding_mask.unsqueeze(-1).to(hidden_states.dtype)
lengths = mask.sum(dim=1).clamp(min=1.0)
mean = (hidden_states * mask).sum(dim=1) / lengths
var = ((hidden_states - mean.unsqueeze(1)) ** 2 * mask).sum(dim=1) / lengths
std = torch.sqrt(var + eps)
return torch.cat([mean, std], dim=-1)
def forward(self, input_values, attention_mask=None, labels=None):
outputs = self.backbone(
input_values=input_values,
attention_mask=attention_mask,
return_dict=True,
)
hidden_states = outputs.last_hidden_state
hidden_states = self.projector(hidden_states)
padding_mask = None
# This is version/model dependent. For Wav2Vec2/WavLM-style models,
# you need the feature-vector-level attention mask, not the raw mask.
if attention_mask is not None and hasattr(self.backbone, "_get_feature_vector_attention_mask"):
padding_mask = self.backbone._get_feature_vector_attention_mask(
hidden_states.shape[1],
attention_mask,
)
pooled = self._masked_mean_std(hidden_states, padding_mask)
logits = self.classifier(pooled)
loss = None
if labels is not None:
loss = nn.CrossEntropyLoss()(logits, labels)
return {"loss": loss, "logits": logits}
For Wav2Vec2/WavLM wrappers, the helper method may live on a nested module depending on whether you are using AutoModel, AutoModelForAudioClassification, or a subclass. So I would treat this as an architecture sketch.
How I would interpret the new results
The new result is encouraging:
| result |
interpretation |
| MFCC/log-Mel + SVM ~90% |
the classes are acoustically separable |
| Wav2Vec2 improved from ~33% to ~40% |
the earlier changes helped |
| still high Lateral recall / low Interdental + Normal recall |
SSL classifier is still collapsing subtle class cues |
| acoustic baseline much stronger than SSL classifier |
representation aggregation is likely the next bottleneck |
So I would not go back to:
maybe the labels are impossible
The acoustic baseline argues against that.
Instead, I would ask:
What does MFCC/log-Mel preserve that the Wav2Vec2/WavLM pooled representation is losing?
My guess: local spectral shape and frame-level variation.
Minimal ablation plan from here
I would keep the next round small and controlled:
| run |
backbone |
pooling head |
purpose |
| A |
current Wav2Vec2/WavLM |
default masked mean |
reproduce current result |
| B |
same |
masked mean + std |
test statistic pooling |
| C |
same |
central mean + std |
test target-focused pooling |
| D |
same |
attention pooling |
test learned frame weighting |
| E |
same |
small TDNN/CNN + stat pooling |
test local temporal pattern head |
| F |
same |
use_weighted_layer_sum=True + best pooling |
test layer selection |
I would not change all variables at once.
First comparison:
default masked mean pooling
vs
masked mean + std pooling
Same:
folds
padding length
fine-tuning setup
learning rates
metrics
If mean+std improves Interdental/Normal recall, the default pooling was probably too lossy.
If mean+std does not help, then I would try central pooling or a small TDNN/CNN head.
Tiny-subset overfit is still useful
Since the pipeline changed, I would still repeat the tiny-subset test.
Example:
20 examples/class
same examples for train and eval
no early stopping
goal: near-100% train accuracy
Interpretation:
| tiny-subset result |
meaning |
| cannot overfit |
training/head/pooling issue still exists |
| can overfit, but CV poor |
generalization / data size / split / regularization issue |
| can overfit only with custom pooling |
default pooling is likely the bottleneck |
| can overfit with MFCC/log-Mel but not SSL |
SSL representation/head path is the issue |
This is still one of the fastest tests for separating “pipeline cannot learn” from “model does not generalize.”
Current best guess
At this point my best guess is:
The task is real and acoustically separable, but the default Wav2Vec2/WavLM sequence-classification head is too coarse for this particular local fricative classification problem.
So my next concrete recommendation would be:
- keep the 0.3s max length for now,
- keep
freeze_feature_encoder() rather than freeze_base_model(),
- use a much lower LR for the backbone than for the head,
- replace default mean pooling with masked mean + std pooling,
- then test central pooling / attention pooling / TDNN head,
- only then tune class weighting or sampling.
Class weights may still help minority recall, but the MFCC/log-Mel result makes me think the next big gain is more likely to come from how the SSL hidden sequence is pooled, not from the loss alone.
Relevant links
, but for now, how about something like this: