That’s true. Real-world cases can reveal things that are easy to miss if you look only at the theory.
I think your example suggests one more boundary that may be worth making explicit: resolving the service concept and deciding actual provider capability are related, but they are not quite the same decision.
A compact version of the pipeline might be:
raw provider wording + context
↓
candidate concepts
↓
concept relation
↓
capability / applicability checks
↓
accept / reject / unsure
That seems especially useful for the kind of cases you pointed to. A label can be semantically close to the right canonical concept while still being a bad operational match because of the object involved, residential vs. commercial scope, geography, eligibility, delivery mode, or some other constraint.
The junk-removal example you linked is actually a nice illustration of this distinction. I would not treat that commercial page as evidence for a particular matching algorithm, but it is useful requirements/stress-case evidence: the provider itself distinguishes items that are accepted normally, accepted with special handling, and not accepted. Even within something as apparently simple as “appliance removal,” washers/dryers and refrigerant-containing appliances have different handling requirements, while other superficially related items are rejected entirely.
So if the goal is actual service capability rather than only semantic discovery, I would probably avoid asking one embedding score to represent all of those distinctions.
The lowest-cost version I can think of would be:
- Keep the original wording and context. Do not overwrite it with the normalized label.
- Use embeddings primarily to retrieve a small candidate set.
- Record the relation to each promising candidate separately — e.g. exact/close, broader, narrower, related, or none/unknown.
- Apply capability checks only where reliable attributes already exist. A structured service-area field can be a filter; a residential/commercial flag can be a feature or filter; missing information can stay unknown rather than being inferred from the service name.
- Allow
UNSURE / no-valid-match. Do not force the nearest candidate to become the canonical answer.
That does not require replacing the current retrieval model. Most of the gain would initially come from separating states that otherwise get compressed into the same similarity number.
There is a useful precedent for the mechanics in the W3C Entity Reconciliation Community Group’s current Reconciliation Service API draft. It keeps a candidate’s score, individual matching features, and the final Boolean match decision separate. Query properties can also be required filters or merely affect ranking, and the spec explicitly notes that some conditions, such as geographic containment, may be naturally binary rather than meaningfully represented by a similarity score.
That is not a service-directory algorithm by itself, but the separation is a useful one here:
retrieval score = "how promising is this candidate?"
relation = "how are these concepts related?"
capability evidence = "does this provider satisfy the relevant constraints?"
final decision = "should this be treated as a usable match here?"
I also think your hard-negative idea becomes even more useful if the test set is treated as a boundary set rather than making every difficult example a negative. Some of the hardest cases are actually positives or partial relations.
For example:
| Case |
What it tests |
| same service, very different wording |
retrieval/paraphrase robustness |
| same or nearly same scope |
equivalence decision |
| broader vs. narrower service |
granularity/relation |
| related sibling services |
relatedness without collapsing |
| strong lexical overlap, different capability |
false-positive resistance |
| same concept, incompatible service area/audience/object |
capability stage |
| no appropriate canonical target |
abstention / forced-match behavior |
| vague bundled record vs. several granular records |
representation/granularity |
That would let errors be assigned to the stage that actually produced them instead of only reporting whether the final pair crossed one threshold.
Why I think granularity is a real part of this problem
There are some surprisingly close examples in the human-service-directory world.
Connect211 and Open Referral recently described an open-source Record Matcher used to compare resource directories. Their problem is entity resolution rather than exactly the service-normalization problem here, so I would not assume the causes are identical. But the service-level failure modes are relevant.
They describe cases where essentially the same cluster of activities is represented as one service record in one directory and five records in another. For service resolution they therefore use multiple kinds of evidence — factual fields, categories/taxonomies, and semantic similarity in names/descriptions — rather than name similarity alone.
That is a useful reminder that a mismatch can come from at least three different places:
different wording
different concept
different record granularity
Those are easy to conflate if all of them ultimately appear as “low/high cosine similarity.”
There is an even more direct granularity discussion in a 2026 Open Referral community thread on describing data quality. One example asks what level should count as the service:
Welfare support
Food bank
Vegetarian food bank
Vegan food bank
Kosher foodbank
Halal foodbank
The point made there is that the useful granularity depends on the user need.
The same discussion gives a nice bundled/unbundled example: a large authority represents a library as one service, while the local township represents the same library as roughly ten to twelve individual services. They argue that preserving granular source information is valuable because reliably aggregating several specific records later is generally easier than reconstructing several specific services from one vague record.
That makes your suggestion to retain the provider’s original wording look important for more than debugging. It helps avoid making normalization an irreversible information-loss operation.
I would therefore be inclined to store something conceptually like:
{
"raw_label": "...",
"raw_context": "...",
"candidate_concepts": [...],
"chosen_relation": "...",
"canonical_concept": "...",
"capability_evidence": {...},
"decision": "...",
"decision_reason": [...]
}
Not necessarily this exact schema — just the separation.
It also means that before changing an embedding model, one cheap sanity check may be to inspect the most frequently confused taxonomy siblings manually:
Can two humans distinguish these concepts from the definitions and available context?
If the boundary itself is unclear, changing the embedding threshold may only move the error around.
Relation types: useful vocabulary without making the ontology too heavy
Your raw label → candidate concepts → relation → canonical service formulation also resembles how some established crosswalk systems separate semantic suggestions from the final relationship.
For example, the European Commission’s ESCO–O*NET crosswalk used transformer embeddings and cosine similarity to generate a ranked list of candidate occupations. Human validators then determined the relationship between the concepts using categories such as exact, broad, narrow, and close; a published variant also includes related mappings.
That is a different domain — occupations are much more standardized than provider-written service descriptions — but the pipeline shape is interesting:
embedding similarity
↓
candidate suggestion
↓
relation validation
rather than:
embedding similarity > threshold
↓
same concept
SKOS supplies related vocabulary:
exactMatchcloseMatchbroadMatchnarrowMatchrelatedMatch
I would use that more as a vocabulary/reference point than as a requirement to implement SKOS itself.
One useful detail is that SKOS deliberately distinguishes closeMatch from identity: it defines it in terms of concepts being sufficiently similar to be interchangeable in some information-retrieval applications. That is already weaker than saying two services have identical operational capability.
So for this use case there may really be two separate questions:
concept relation:
exact / close / broader / narrower / related / none
operational applicability:
yes / no / conditional / unknown
For example:
"refrigerator removal"
concept relation to "appliance removal":
narrower
provider capability:
conditional (special refrigerant handling)
or:
"biohazard cleanup"
concept relation to "junk removal":
related
provider capability:
no
That seems more faithful than trying to encode both facts into a single “0.78 similar” value.
I would also keep confidence scoped. A retrieval score, confidence in the concept relation, and confidence in the final capability decision are not necessarily the same quantity.
Something like:
retrieval_score
relation
relation_confidence
capability_status
final_decision
is easier to interpret later than a single generic confidence = 0.82.
Again, I would not necessarily implement all of those fields on day one. The main value is keeping the meanings distinct.
A cheap capability layer: structured first, extraction later
I think this is where the real-world cases are particularly useful.
A reasonable default could be:
Is the condition already structured?
|
+-- yes --> use it directly as a filter/feature where appropriate
|
+-- no --> is it reliably present in free text?
|
+-- yes --> preserve the text; optionally extract later
|
+-- no --> leave capability UNKNOWN
For example:
service_area structured?
-> cheap deterministic geographic check
residential/commercial structured?
-> cheap compatibility feature/filter
age/eligibility structured?
-> cheap compatibility check
accepted/excluded object explicitly structured?
-> cheap capability check
only implied by marketing copy?
-> do not pretend the canonical label answered it
I would resist making every contextual field a hard gate. Some attributes are hard constraints, while others should merely influence ranking.
The W3C reconciliation draft has a nice generic distinction here: a property can be marked required, in which case it acts like a filter, or optional, in which case it affects the candidate ordering. Its matchQualifier mechanism can also represent different relations rather than assuming equality.
The practical service version might be:
HARD / near-hard
- outside service area
- explicit "commercial only"
- explicit exclusion
- eligibility definitely not satisfied
SOFT
- preferred specialization
- distance
- likely but not guaranteed scope
- customer wording similarity
UNKNOWN
- information not present
The key point is that unknown should not silently turn into either true or false.
That is also why I would start with attributes the data already contains. Trying to immediately extract every capability dimension from arbitrary provider prose with an LLM would turn a relatively small architecture improvement into a second large NLP problem.
If later error analysis shows that one missing modifier is responsible for many failures — e.g. residential/commercial or emergency/non-emergency — then extracting that one field becomes a much more targeted investment.
Evaluation: separate retrieval failures from decision failures
The public Connect211 training/tuning notes are useful here because they explicitly diagnose errors by pipeline stage.
Their exact task is duplicate/entity resolution, so the labels are not directly transferable, but the diagnostic idea is.
They distinguish, for example:
not a candidate
-> blocking/retrieval failure
candidate, but below decision threshold
-> scoring/decision failure
candidate scored sufficiently, but later vetoed
-> mitigation/gating failure
They also recommend building labeled ground truth before parameter tuning, sample different score bands, inspect near-misses below the threshold, and change one variable at a time.
For the service-matching pipeline, I think the analogous evaluation could be quite small:
1. Candidate retrieval
Question:
Did the appropriate concept or useful related concept survive into top-k?
Possible metrics:
Recall@1
Recall@5
Recall@10
The important metric is probably not “average cosine similarity.” It is whether a correct/useful candidate was available to the next stage.
2. Relation classification
Given the relevant candidate, did the system call it:
exact
close
broader
narrower
related
none
correctly enough for the downstream use?
A confusion matrix here would reveal something that a single global matching accuracy hides — e.g. perhaps exact vs. related is the real problem.
3. Capability/applicability
Given an acceptable concept relation, did contextual evidence reverse the operational decision when it should?
For example:
same broad service concept
+ wrong geography
= do not return as capable provider
same broad service concept
+ explicitly excluded object
= do not return as capable provider
narrower service
+ compatible request
= may be an excellent match
4. Abstention
Include cases where the correct answer is:
no canonical target
insufficient evidence
needs review
Then measure how often the system forces one of them into the nearest known concept.
This seems particularly important for semantic embeddings because there is always a nearest vector even when no good semantic answer exists.
5. Boundary set
Rather than immediately creating a huge benchmark, a manually reviewed set of perhaps a few dozen recurring boundary cases may tell you much more.
I would include at least:
A. Same concept / very different customer wording
B. Same concept / one extra modifier
C. Broader vs. narrower
D. Related siblings
E. Heavy lexical overlap / genuinely different service
F. Same concept / incompatible capability
G. Bundled vs. granular record
H. No valid target
Then each future model/threshold/policy change can run against the exact same cases.
That turns “this embedding model feels better” into something much easier to debug.
One caution on hard-negative mining
I agree with the direction of your hard-negative suggestion, but there is one trap I would explicitly guard against: a semantically close neighbor is not automatically a negative.
That is especially dangerous in a hierarchy.
Suppose the taxonomy contains:
cleanup
├── estate cleanup
├── construction cleanup
└── biohazard cleanup
If a training script blindly takes the nearest sibling as a hard negative, some examples may actually be valid broader/narrower/related matches depending on the request and intended output.
There is a recent analogous example in the paper Fine-Grained Curriculum Standards Alignment on the MathFish Benchmark. The task is education rather than services, but the authors specifically note that sibling standards sampled as hard negatives can occasionally be true positives, which can teach the bi-encoder to over-suppress genuinely related standards.
So I would probably mine difficult neighbors automatically, but review the boundary labels before using them as negatives.
In other words:
nearest difficult neighbor
↓
manual/reliable relation label
├── exact/close -> positive
├── broader/narrower -> structural relation
├── related -> related case
├── capability-only mismatch
└── true negative
This is one reason I like “boundary set” as the broader evaluation concept. Not every useful difficult pair needs to become a contrastive negative.
What I would probably not change first
Given the architecture you described, I would not start by:
- fine-tuning a new embedding model,
- building a large custom ontology,
- adding an LLM judge to every pair,
- extracting every possible capability field,
- or searching for one globally optimal cosine threshold.
Those may eventually be useful, but they make several variables move at once.
The cheaper sequence seems more diagnostic:
1. Preserve raw wording/context.
2. Freeze the current retriever as a baseline.
3. Save top-k candidates and retrieval scores.
4. Add a small relation label.
5. Add only capability checks supported by reliable existing fields.
6. Permit UNKNOWN / UNSURE.
7. Build a small real-world boundary set.
8. Measure which stage actually fails.
9. Only then decide whether the retriever, relation model,
taxonomy, capability extraction, or decision policy needs work.
This also gives a cleaner answer when something fails.
Instead of:
“the semantic matcher got this wrong”
you can say:
the right concept never entered top-k
or:
retrieval was fine; the system collapsed a related concept into exact
or:
the canonical concept was fine, but a capability constraint was missing
or:
the source record itself was too vague to decide
Those point to very different fixes.
One thing I found especially useful in the real-world directory examples is that they make the normalization problem look less like “find the perfect canonical string” and more like “preserve enough distinctions that later stages can make the decision appropriate to their use case.”
So I think your original pipeline is still a good core:
raw label
-> candidate concepts
-> relation
-> canonical service
I would just avoid treating canonical service as the end of the operational decision when capability matters.
For a discovery/search use case, a related or broader match may be perfectly useful.
For an actual provider-capability decision, the system may need one more layer:
canonical concept
+ provider/request context
-> applicable / not applicable / conditional / unknown
That distinction seems to preserve the useful part of semantic normalization without asking normalization to erase exactly the real-world details that made the service distinguishable in the first place.