Hmm… if the subject is social science, an absolute “right answer” is probably infinitely far away
, but putting that aside, I wonder if this could be improved quite a bit by stress-testing the classifier more aggressively?
From the public description alone, BERTopic + a RoBERTa model trained on GoEmotions seems like a perfectly reasonable exploratory V1 stack to me. I would not start by replacing either model.
The place I would stress-test first is the measurement path between:
text
→ classifier scores
→ emotion labels / probabilities
→ aggregation
→ post-vs-comment differences / temporal trends
→ substantive interpretation
That seems especially important here because your questions are not only “does the classifier label individual documents reasonably?”, but also “do these labels support aggregate patterns across document types and time?”
Those are related questions, but not quite the same one.
If I wanted the highest-information checks for the least additional work
My default order would probably be:
-
Freeze and document the exact emotion inference contract
checkpoint/revision → label mapping → multi-label rule → thresholds → aggregation rule → denominator.
-
Perturb the emotion measurement without changing the model
and see whether the headline findings survive:
- threshold sensitivity;
- thresholded labels vs probability aggregation, where both are meaningful;
- post vs comment;
- early/middle/late time slices;
- short vs long/truncated text;
- especially labels such as fear/nervousness if they carry much of the interpretation.
-
Use a strong LLM as an independent second annotator on a stratified sample, blinded to the GoEmotions outputs.
-
Send only disagreements / unstable cases / headline-critical cases to a small blind human audit if manual labeling is expensive.
-
Only escalate to model replacement, recalibration, larger annotation, or more formal correction if one of those tests actually moves the conclusion.
So rather than making V1 much bigger immediately, I would use stress tests as a gate: if the findings are stable, stop; if something moves, that result tells you exactly where further work is worth spending time.
There is also one scope distinction that I think could make the interpretation easier rather than weaker:
observed Reddit text
→ model-detected / text-expressed emotion
→ climate-related affective discourse
→ climate-anxiety construct
→ individual psychological state
→ wider/youth population state
V1 can potentially be quite useful on the left side without having to prove every bridge on the right side. I would treat those later bridges as separate validation problems rather than as reasons to discard the current analysis.
Why I would stress-test the emotion measurement before changing the classifier
Document classification and aggregate measurement are different targets
GoEmotions is particularly attractive here because it was itself built from English Reddit comments and is explicitly multi-label: 27 emotion categories plus neutral.
But once classifier outputs are used to estimate quantities such as:
- “X% of texts expressed fear,”
- “comments contain more fear than posts,”
- “fear increased over time,”
the target has changed from individual-document classification to quantification / downstream statistical estimation.
A classifier can therefore be reasonably good at individual predictions while still shifting an aggregate quantity enough to matter.
This is not specific to LLMs. It is the broader “predicted variable” problem. Recent work in computational social science makes the point quite sharply: automated labels can induce biased downstream estimates and invalid confidence intervals even when the label predictor itself has high accuracy. A useful recent overview/framework is Egami et al., Using Large Language Model Annotations for the Social Sciences. If formal statistical correction eventually becomes important, Prediction-Powered Inference is one related general framework.
A small proxy sanity check
I tried a small public proxy experiment with SamLowe/roberta-base-go_emotions on the standard GoEmotions validation/test data.
This is not your model unless you happen to use that exact checkpoint, and not your climate corpus, so I would not interpret it as evidence that your results have this problem. I only wanted to see whether the failure mode was large enough to care about.
It was.
For example, for fear, the true test prevalence in that split was about 1.44%. Depending on the threshold used in the proxy run, predicted prevalence ranged roughly from 0.61% to 2.54%.
More interestingly, choosing thresholds to improve validation-set document-level F1 did not automatically improve aggregate prevalence estimation on the test set.
In that particular proxy:
| Aggregation method |
Mean absolute prevalence error across labels |
| Mean predicted probability |
~0.0036 |
| Top-1 |
~0.0071 |
| Fixed 0.5 threshold |
~0.0114 |
| Per-label threshold tuned for validation F1 |
~0.0130 |
I would not conclude that probability averaging is therefore universally the right estimator; the correct estimator depends on what quantity you mean to estimate and on calibration. The useful point is narrower:
optimizing the classifier for document-level F1 and preserving aggregate emotion prevalence are not necessarily the same objective.
There is also substantial label-to-label heterogeneity. The public model card itself shows very different operating characteristics across emotions and provides per-label threshold results rather than suggesting that every label behaves equivalently.
So if some of the main substantive conclusions depend heavily on a small set of labels, I would probably inspect those labels individually rather than relying only on one macro metric.
A very cheap robustness table could already tell a lot
For each headline emotion/contrast, something like this would be surprisingly informative:
| Result |
Original pipeline |
Lower threshold |
Higher threshold |
Probability aggregation |
Direction stable? |
| fear: posts vs comments |
… |
… |
… |
… |
yes/no |
| sadness: posts vs comments |
… |
… |
… |
… |
yes/no |
| fear: early vs late |
… |
… |
… |
… |
yes/no |
You would not need to claim that every column is equally “correct.” It is a sensitivity analysis.
If the sign and rough magnitude survive reasonable alternatives, the original finding becomes easier to trust.
If they do not survive, that is also useful because you have localized the uncertainty to the measurement layer rather than having to question the entire project.
A relatively low-HITL validation route using a strong LLM
If a conventional human annotation exercise is too expensive for V1, I think current LLM-annotation work gives you a useful intermediate option.
I would use a strong LLM not as the new ground truth, but as an independent second measuring instrument.
Something along these lines:
stratified sample
(post/comment × time × important emotion)
│
├── GoEmotions classifier
│
└── strong LLM
blinded to GoEmotions output
│
↓
compare annotations
│
┌───────────┴───────────┐
│ │
agreement disagreement
│
↓
small blind human audit
The LLM could also annotate distinctions that a generic emotion label alone cannot expose very well.
For example:
{
"expressed_emotion": ["fear"],
"experiencer": "self | other | generic | unclear",
"mode": "direct | reported | quoted | sarcastic | unclear",
"target": "climate | politics | other | unclear",
"climate_anxiety_evidence": "yes | no | ambiguous"
}
The exact schema is not important; the important part is separating cases such as:
"I'm terrified about what climate change will do here."
from:
"The article says residents are terrified."
or sarcastic/quoted/negated cases.
That would tell you whether a “fear” trend is really a change in first-person affective expression, a change in reporting about fear, a topic-composition effect, etc.
There is increasingly good evidence that strong LLMs can be useful in this role. For example, Ruder et al. (CLPsych 2025) found GPT-4 competitive with human reader-annotators on emotion-appraisal annotation, with five-completion majority voting improving reliability. Interestingly, they also found that more complicated instructions could make performance worse, so I would start with a small clear rubric rather than a giant ontology.
If you eventually want a formal justification for replacing much of the human annotation, the Alternative Annotator Test (ACL 2025) is almost exactly about that question: using a modest human-labeled subset to test whether an LLM can defensibly serve as an alternative annotator for the task.
There is also a particularly nice middle ground for computational social science: Confidence-driven inference (NAACL 2025) uses LLM confidence information to choose which examples actually need human labels. Across the three social-science tasks in that paper, it reduced required human annotation by more than 25% while retaining statistical validity guarantees.
So there is a continuum rather than a binary choice:
all human
↑
stratified blind human audit
↑
human review only for LLM-uncertain/disagreement cases
↑
strong LLM second annotation
↑
existing classifier only
For a V1, I would probably start near the bottom and move upward only as far as the observed disagreements justify.
One HITL trap I would avoid
I would not build the human validation set by first displaying the LLM answer and asking a human to approve/reject it.
Schroeder et al., Just Put a Human in the Loop? (Findings ACL 2025) ran a preregistered study with 350 annotators / 7,000 annotations and found that showing people LLM-generated suggestions significantly shifted the human label distribution.
For a subjective task such as emotion, I would therefore keep at least the small human anchor blind to both the LLM and GoEmotions prediction.
That gives you an independent reference rather than a “human-approved model label.”
Post vs comment and temporal comparisons are the slices I would test first
These seem especially interesting in your project, so I would protect rather than de-emphasize them.
The issue is just that:
using the same classifier everywhere does not guarantee equal measurement error everywhere.
Posts vs comments
Posts and comments can differ in:
- length;
- truncation probability;
- conversational dependence;
- quotation/reporting;
- directness of emotional expression;
- subreddit/topic mixture.
So a small table stratified by post/comment could be more informative than another global classifier benchmark.
For example:
| Diagnostic |
Posts |
Comments |
| median tokens |
… |
… |
| % truncated |
… |
… |
| GoEmotions ↔ LLM disagreement |
… |
… |
| fear prevalence, original rule |
… |
… |
| fear prevalence, alternate rule |
… |
… |
If the post/comment contrast remains after those perturbations, that is a stronger result.
Context may also be worth a tiny test rather than a pipeline rewrite. Conversation-emotion research has repeatedly found that surrounding context can affect utterance-level emotion classification; see for example Ghosal et al., Findings ACL 2021.
For a sample of comments, one inexpensive comparison could therefore be:
comment alone
vs
parent + comment
If predictions barely move, great — no need to complicate the main pipeline.
If they move systematically, you have identified a specific limitation rather than a generic “context might matter” caveat.
Temporal results
For temporal trends I would do the same thing:
early period
middle period
late period
and compare:
- document count;
- post/comment mixture;
- subreddit/topic mixture;
- text length/truncation;
- classifier ↔ independent-annotator disagreement;
- headline emotion under alternate thresholds/aggregation.
Language and platform discourse can drift over time. That does not establish that your classifier has drifted, but temporal degradation of social-media classifiers is a known enough phenomenon that a small diagnostic is justified; VIBE (EMNLP 2023) is one example studying temporal deterioration/adaptation in Twitter classification.
Again, I would not build an elaborate temporal-adaptation model unless the cheap check actually shows a problem.
If the trend survives the cheap check, the result gets stronger.
If it weakens, you learn whether the likely source is measurement drift or corpus-composition change.
Both outcomes are informative.
For BERTopic, I would test substantive robustness rather than search for a 'correct' parameter set
BERTopic itself would be lower on my concern list than the emotion measurement.
I would separate reproducibility from robustness here.
The BERTopic FAQ and best-practices documentation explicitly note that default UMAP behavior is stochastic; setting random_state can make a run reproducible.
But:
same seed gives same result
answers a different question from:
would the substantive topic story survive another reasonable specification?
For the latter, I would only run a small perturbation grid.
For example:
baseline
+ 2–3 UMAP seeds
+ low / baseline / high n_neighbors
+ low / baseline / high min_cluster_size
Maybe one alternative embedding model if embedding choice is itself central to the paper.
I would not require exact topic IDs to remain stable. I would compare things a reader actually cares about:
- major topic top terms;
- representative documents;
- overlap/alignment of major clusters;
- prevalence ranking of major topics;
- whether the interpretation of the headline topics changes.
If the main topic structure survives a handful of reasonable perturbations, I would stop there rather than turning V1 into a topic-model hyperparameter paper.
If one major interpretation disappears whenever one reasonable parameter changes, that is the one branch worth examining.
A scope separation that might help with the climate-anxiety interpretation
This is the part where social science becomes much harder than classifier benchmarking, so I would keep the layers explicit rather than trying to solve all of them simultaneously.
I see roughly:
Reddit text in the collected corpus
↓
emotion expressed/detected in that text
↓
climate-related affective discourse
↓
climate-anxiety construct
↓
psychological state of the author
↓
state of a target population such as young people
↓
support/intervention decision
The farther right you go, the more additional assumptions and validation are needed.
That does not mean the left side is uninteresting.
If anything, I think separating the stages lets V1 make a cleaner contribution:
describe topic structure and model-detected/text-expressed emotional patterns in the analyzed climate-related Reddit discourse
and then explicitly make something like:
validate the connection between these textual indicators and climate-anxiety measures in an appropriate target population
a later research stage.
That seems preferable to forcing a generic emotion classifier to carry the entire psychological construct.
This distinction is part of a wider measurement-validity issue in computational social science. Baden et al., Three Gaps in Computational Text Analysis Methods for Social Sciences is a useful discussion of why predictive/technical validity and social-scientific measurement validity need not coincide.
The same separation becomes important if the ultimate target is broader than “the Reddit discourse in this dataset.” A useful framework for thinking through the additional sampling/platform/population steps is the Total Error Framework for Digital Traces of Human Behavior on Online Platforms.
For example:
number of documents ≠ number of people
Reddit users ≠ a population sample
one subreddit ≠ Reddit
Reddit ≠ young people in general
and heavy users, large threads, subreddit composition, deletion/availability, etc. can all affect corpus-level trends.
But I would treat most of this as scope documentation / future validation, not as a demand to solve everything before V1 is publishable.
The cheap classifier stress tests above come first because they can tell you whether there is even a measurement problem worth spending more resources on.
So, in compact form
If the manuscript does not already contain equivalent checks, my first pass would be:
1. Keep BERTopic + GoEmotions.
2. Make the exact inference/aggregation rule reproducible.
3. Stress-test headline emotion results against reasonable thresholds/aggregation.
4. Check post/comment and early/middle/late slices separately.
5. Run a strong LLM as a blinded independent annotator on a stratified sample.
6. Human-review only a small blind sample, emphasizing disagreements/critical cases.
7. Perturb BERTopic just enough to test whether the substantive topic story survives.
8. Escalate only where one of those checks changes the conclusion.
That seems to me like a relatively inexpensive way to get a lot more confidence in the current V1 without turning it into a completely different project.
And importantly, a stress test does not only have value when it “passes.” If a headline result changes under one perturbation, that gives you a useful design boundary:
stable under A/B/C
sensitive to D
which is much more actionable than a generic limitation.
Since I have only seen the public project description rather than the manuscript/methods/code, I would phrase all of this conditionally — some or all of these controls may already be in the preprint. But given the three areas you explicitly asked about — modelling choices, emotion methodology, and interpretation — the classifier-output → aggregate-result path is the first place I would spend validation effort before considering a larger redesign.