For now, here’s what I found from some light testing in Colab:
I tried to reproduce the public numbers first, then deliberately looked for easy failure modes rather than assuming the headline accuracy was telling the whole story.
The short version is: the basic result held up better than I expected.
On the public 80/20-style split I reconstructed, I got:
- TF-IDF / Logistic Regression fast path: 121/146 = 82.88%
- published neural model: 145/146 = 99.315%
That is broadly consistent with the results described in the repository and model card.
I also tried a stronger grouped holdout where related lexical families were kept together rather than randomly split. The neural model was still essentially unchanged:
- ordinary IID split: 99.50%
- lexical-family grouped split: 99.78%
So I didn’t find evidence for the simple explanation that the reported neural accuracy is mostly coming from generic near-duplicate/template leakage.
The more interesting signal was narrower: L3 / Predictive seems to have much less diversity in how its core temporal relation is expressed than L2 or L4.
If I were extending the evaluation, my default route would therefore be quite small:
- keep the current 4-layer model;
- add a tiny author-reviewed semantic/paraphrase holdout, especially for L3;
- define what should happen when one utterance contains signals from more than one layer;
- report multilingual and fast-path results with a little more provenance.
That would probably tell more than simply adding another large random test split.
1. What reproduced, including the negative result
Baseline reproduction
The public dataset has 730 examples across the four layers and nine languages. The repository describes an 80/20 split for the fast classifier, and I was able to reproduce essentially the published fast-path result:
| model |
result I got |
| TF-IDF + Logistic Regression |
121 / 146 = 82.88% |
| published DistilBERT model |
145 / 146 = 99.315% |
That made me less interested in trying to explain the result away through a basic train/test bug.
I then tried a grouped split designed to make the test slightly less friendly: examples belonging to the same obvious lexical family were kept on the same side of the split.
The result was:
| neural evaluation |
accuracy |
| ordinary IID split |
99.50% |
| grouped lexical-family holdout |
99.78% |
The exact numbers should not be overinterpreted because this is still a small dataset, but the useful part is the negative result: the neural classifier did not collapse when the obvious lexical families were separated.
So I would not characterize this as a generic “keyword-only” or “template memorization” result.
That also matches another diagnostic I tried later: several L2/L4 trigger families remained strong even when that family was held out during training.
2. The narrower signal: L3 paraphrase coverage
L3 looks structurally different from L2/L4 in the English training support
Looking through the English portion, L2 and L4 have several different surface forms for their defining relations.
L2, for example, has conditional/temporal constructions around things such as:
L4 similarly has multiple recurring autonomous/persistent forms.
For English L3, however, I found that 99/99 examples contain before.
I did not find comparable training support for expressions such as:
ahead ofin advance ofin time forby the time- constructions where the same anticipatory relation is expressed without the word
before
That observation by itself does not show that the neural model is simply matching before. In fact, the other probes give evidence against such a broad conclusion.
It means something more limited: the dataset gives the model much less evidence about alternative linguistic realizations of the L3 relation.
There is also a practical evaluation problem here: a normal “leave the before trigger out” experiment is not very meaningful, because removing that family removes essentially the entire English L3 training class.
Small paired paraphrase canary
I therefore tried something weaker and more diagnostic: small paired cases where the underlying intent was intended to stay constant but the obvious surface expression changed.
Across the 22-case canary, the published neural model gave:
| group |
canonical |
paraphrased |
| L2 |
6/6 |
3/6 |
| L3 |
6/6 |
0/6 |
| L4 |
6/6 |
6/6 |
| overall |
20/22 |
13/22 |
I would not call 13/22 or 59.1% a benchmark accuracy. These were hand-written diagnostic cases, not an independently sampled test set.
What caught my attention is the asymmetry: L3 was perfect on the canonical cases and zero on that small paraphrase set, while L4 remained intact.
Useful disconfirming evidence
I also tried held-out trigger-family tests where there was enough remaining training support to make the experiment meaningful.
For the neural model I got roughly:
| held-out family |
accuracy |
whenever |
80.0% |
automatically |
83.9% |
continuous |
93.3% |
always |
94.1% |
every |
100% |
once |
100% |
So again, I don’t think “the model only memorized trigger words” is a good summary.
The narrower hypothesis seems more plausible to me:
L3 may currently have a semantic-support problem rather than a general lexical-memorization problem.
A cheap next test would be a small frozen set of author-reviewed L3 paraphrases before adding those paraphrases to training.
Something like 20–50 examples is probably already informative if they deliberately cover different realizations of the same anticipatory relation.
This is also a fairly standard robustness concern for intent classifiers. For example, Sengupta et al. tested production-like noise including paraphrases and synonyms and found that these perturbations can materially affect BERT-based intent classification: On the Robustness of Intent Classification and Slot Labeling in Goal-oriented Dialog Systems to Real-world Noise.
The nice thing here is that this does not require changing the architecture. If the small holdout passes, great; if it fails, a few additional semantic families in the training data may be enough.
3. Mixed semantics: I think the routing contract matters as much as the classifier
The four layers are interesting because they are not just conventional topical intents; they imply different execution semantics:
- L1 → immediate
- L2 → conditional
- L3 → anticipatory
- L4 → persistent/autonomous
That creates one boundary case that seems worth specifying explicitly: what happens when one command legitimately contains more than one of those signals?
For example:
“Always prepare the renewal pack before the contract expires.”
There is an L4-style persistent instruction and an L3-style anticipatory relation in the same utterance.
Likewise:
“Whenever a customer reaches the renewal window, prepare the account review before their contract expires.”
That can plausibly contain both L2 and L3 structure.
This does not necessarily mean the classifier should become multi-label. If downstream execution requires one routing class, keeping a single label may actually be preferable.
I can see at least two clean designs.
Option A — keep exactly one layer and define precedence
For example, if the intended semantics are something like:
L4 > L3 > L2 > L1
then that becomes part of the classifier contract.
Mixed examples can then be included explicitly in the evaluation:
persistent + predictive -> ?
conditional + predictive -> ?
persistent + conditional -> ?
The useful thing is not any particular precedence I would choose; it is making the intended precedence explicit enough that the gold label is reproducible.
Option B — separate routing from semantic attributes
Another possible decomposition would be:
route = L1/L2/L3/L4
attributes:
conditional = true/false
anticipatory = true/false
persistent = true/false
The route can remain a single four-way classification while the secondary semantics are preserved for the executor.
That may be overkill if the current downstream system only needs one label, though.
There is broader work on multi-intent utterances showing why single-intent assumptions can become awkward for naturally blended user requests; BlendX is one useful reference. I would not take that as an argument that ConceptNet itself needs multi-label classification — only as background for why explicitly defining the mixed case can save ambiguity later.
For this project, a documented precedence rule plus 10–20 mixed examples may be enough.
4. Multilingual evaluation: per-language numbers and provenance would help
The multilingual part is potentially useful, but with 730 examples the pooled score can hide quite different amounts of evidence per language.
The current repository dataset table has:
| language |
examples |
| English |
400 |
| French |
45 |
| Spanish |
45 |
| German |
40 |
| Italian |
40 |
| Portuguese |
40 |
| Chinese |
40 |
| Arabic |
40 |
| Russian |
40 |
So an 80/20 evaluation leaves only a handful of examples for some individual languages.
That is not necessarily a problem for an early dataset, but I would interpret a pooled “9-language” accuracy differently from a mature per-language benchmark.
A very cheap reporting improvement would be something like:
language | test n | accuracy | macro-F1
and, where n is tiny, just leave it visible rather than drawing strong conclusions from the percentage.
One provenance detail that may be worth documenting
In the current public dataset, the Russian examples I checked are represented in Latin transliteration rather than Cyrillic.
There are at least two reasonable interpretations:
If romanized/ASR-style Russian is the intended input:
then this may be exactly what the system wants; I would simply document that input assumption.
If native-script Russian is also a deployment target:
a small Cyrillic holdout would give useful information at very low cost.
More generally, it would help future users if the dataset card said how the non-English examples were produced:
- native authored;
- professionally localized;
- machine translated;
- LLM generated/translated;
- human reviewed;
- or some combination.
This is mostly about making the evaluation interpretable rather than requiring a particular methodology.
For comparison, MASSIVE is much larger and has a different purpose, but one thing it does particularly well is describe how the English source utterances were localized into the other languages. That kind of provenance makes multilingual results much easier to reuse.
5. Fast path: separating accuracy from coverage would make the cascade easier to read
One small terminology point may be worth cleaning up because the architecture itself is reasonable.
The forum post and HF model card describe the fast classifier as roughly 83% accuracy.
The current GitHub architecture diagram describes the same fast path as 83% coverage, followed by neural fallback when confidence is below a threshold.
Those are two different quantities.
For a cascade, I think the most informative reporting would be:
threshold
fast-path coverage
fast-path accuracy on accepted examples
fallback rate
final end-to-end cascade accuracy
I did a small threshold sweep just to see what that presentation might look like. This is only an illustrative diagnostic on the public data, not a claim about the production cascade:
| threshold |
fast coverage |
fast accepted accuracy |
final cascade accuracy |
| 0.50 |
69.2% |
95.0% |
95.9% |
| 0.55 |
60.3% |
98.9% |
98.6% |
| 0.65 |
43.2% |
100% |
99.315% |
That exposes the actual deployment trade-off very directly:
How many requests can the cheap classifier safely accept before falling back?
This is closely related to selective prediction / abstention. A useful reference is The Art of Abstention: Selective Prediction and Error Regularization for NLP, which also discusses classifier cascades as an accuracy/efficiency trade-off.
So I would probably keep the two-stage architecture and just make coverage and accuracy separate fields.
6. A small metric/checkpoint provenance table could remove several ambiguities
There are currently a few different numbers visible across the public materials:
- neural accuracy: 98.6%
- fast-path accuracy: about 83%
- L3 Predictive precision: 100%
- my evaluation of the published neural weights on the reconstructed public test split: 145/146 = 99.315%
I don’t think these necessarily conflict. They appear to be attached to different classifiers/checkpoints/evaluation states.
For example, the repository README shows L3 precision 1.00 inside the fast-path classification report, while the model card lists “L3 Predictive precision: 100%” alongside the neural headline without identifying the source of that number.
Likewise, the published neural weights reproduced the higher evaluation result for me, whereas the 98.6% headline appears to correspond to another point in the training/evaluation history.
A tiny provenance table would make all of this self-explanatory:
| reported result |
classifier/checkpoint |
split |
metric |
| ~83% |
TF-IDF/LR |
public 80/20 |
accuracy |
| 100% L3 |
TF-IDF/LR |
same split |
precision |
| 98.6% |
neural |
specify checkpoint |
accuracy |
| 99.315% |
published neural weights |
specify/reconstructed split |
accuracy |
The main benefit is reproducibility: a future reader immediately knows which number belongs to which stage.
7. Two tiny repository wiring notes
These are separate from the model results, but I hit two small reproducibility nits while following the current README.
The documented quick start is:
git clone <repository>
cd ConceptNet
pip install scikit-learn
python core/train_classifier.py
The dataset is stored at:
data/conceptnet_dataset_v2.json
but the current training script opens:
with open("conceptnet_dataset_v2.json", "r", encoding="utf-8") as f:
The neural script similarly opens the filename relative to the working directory.
So running the commands literally from the repository root can miss the dataset unless it has also been copied there.
Using a path relative to the script/repository root would make the quick start location-independent, for example conceptually:
repo_root = Path(__file__).resolve().parent.parent
dataset_path = repo_root / "data" / "conceptnet_dataset_v2.json"
I would treat this purely as a packaging/documentation fix, not a modeling issue.
There also appears to be a little API/core drift in the current tree, so a fresh-clone smoke test such as
git clone ...
pip install -r requirements.txt
python core/train_classifier.py
# start API / run one known request
in CI would probably catch these cheaply.
Overall, the part I found most encouraging was actually the failed attempt to break the neural result with a generic lexical-family split. The obvious “99% only because the random split leaked the templates” explanation did not survive that check.
The next useful question therefore seems narrower: whether the four execution semantics remain stable when the same relation is expressed differently, and when multiple relations appear in one utterance.
If this were my evaluation budget, I would spend the next small increment on:
- a frozen, author-reviewed L3 semantic paraphrase holdout;
- a tiny mixed-semantics / precedence set;
- per-language counts and scores;
- explicit cascade coverage-vs-accuracy reporting.
Those are all relatively cheap, and each one answers a different deployment question without requiring a redesign of the core model.
conceptnet.co.uk/sandbox/
L4 > L3 > L2 > L1 makes sense as a routing contract. I’ll document that explicitly.