Hmm… without measurements from the actual environment, it may be hard to say where the bottleneck really is:
At a high level, your pipeline looks reasonable to me. In entity-resolution terminology, it is roughly:
blocking / candidate generation
-> pairwise matching / verification
-> downstream presentation or clustering
Using embeddings for high-recall candidate generation and a stronger second-stage verifier is a common pattern. Splitting a large embedding workload across two T4s is also a valid scaling method; Sentence Transformers supports multi-GPU encoding directly.
The part I would keep separate is:
- Is the matching pipeline itself efficient and accurate?
- How should that pipeline be scheduled across the two GPUs?
Two T4s give you more total compute, but they do not by themselves tell you whether embedding is the actual end-to-end bottleneck, whether each GPU should be dedicated to a different model, or whether the cloud LLM is where most time and cost are going.
My default path would probably be:
high-recall candidate generation
-> small task-specific local pair verifier
-> confident match / non-match: resolve locally
-> uncertain, shifted, or disputed cases: cloud LLM
-> reviewed / corrected outcomes: future verifier training data
This avoids making the decision “LLM for every pair” versus “no LLM at all.” The LLM can remain useful as an escalation layer, a source of provisional labels, and a way to handle new schemas or unfamiliar cases, while repetitive in-distribution pairs move to a much cheaper local classifier.
For the four questions in the post, my provisional answers would be:
| Question |
What I would try first |
| Embeddings for short/noisy text |
Evaluate them as blockers, using candidate recall at a fixed candidate budget. Preserve field names, identifiers, numbers, units, and missingness instead of flattening everything into generic prose. |
| LLM verifier reliability |
Compare the current LLM with a task-specific pair classifier trained on the actual false-positive tail. Keep the LLM for uncertain or shifted cases. |
| Two T4s |
Treat two-GPU embedding as a valid baseline, not an automatically optimal deployment. Also test one-GPU co-location and flexible two-GPU scheduling. |
| Similar-but-different false positives |
Turn those production errors into named hard-negative slices: model-number conflicts, variants, versions, quantities, granularity differences, and insufficient evidence. |
Before changing the architecture, I would instrument one representative scheduled scan and collect only a small set of counts and timings:
Counts
- records scanned
- records changed since the previous scan
- total input tokens
- candidate pairs generated
- candidates per record
- local verification calls
- cloud verification calls
- cloud input/output tokens and cost
Stage timings
- source read / parsing / normalization
- tokenization and padding
- embedding forward pass
- index search / metadata filtering / top-k
- candidate-pair construction
- local verifier
- cloud queue / request / generation / parse / retry
- result serialization / dashboard handoff
Resources
- CPU utilization per core
- process / thread / tokenizer-worker counts
- RAM and I/O
- utilization and VRAM for each T4
- queue depth and GPU idle gaps
That should distinguish several very different situations:
Both T4s busy, embedding dominates
-> embedding scaling and model choice are worth optimizing
T4s frequently idle, CPU cores busy
-> tokenization, normalization, pair construction, or thread contention
T4s and CPU both idle
-> index, I/O, queueing, synchronization, network, or cloud wait
Embedding is fast but candidate count is large
-> blocking quality may matter more than GPU throughput
Cloud verification dominates
-> local verifier + selective escalation is likely the larger lever
I also ran two small public controls—not as a reproduction of your system, but to check whether the proposed alternatives were mechanically plausible.
On a public Walmart–Amazon entity-matching split, a small task-specific DistilBERT pair classifier substantially outperformed generic semantic-similarity controls on the “very similar but not the same entity” tail. On a Colab T4, a representative MiniLM embedder and DistilBERT verifier also coexisted using only a small fraction of the card’s 16 GB VRAM. In that control, isolated GPU forward throughput was noticeably higher than end-to-end throughput, and coarse GPU utilization during the verifier pipeline averaged only around the middle of the available range on a two-core host.
Those measurements do not establish what will happen on your models or data, but they are enough to make these two comparisons worth including:
- a task-specific local verifier versus generic semantic similarity;
- one-card co-location versus permanent model-per-GPU specialization.
1. Define the operating point before comparing models
The best model and threshold depend heavily on what happens after a positive decision.
If a positive pair is automatically merged
False positives can be much more expensive than false negatives, especially if one erroneous edge merges two larger clusters. Precision on hard negatives may matter more than aggregate F1.
If matches are shown for human review
Candidate recall, review volume, ranking quality, and time saved per reviewer may be more useful than a single binary F1 score.
If every pair remains independent
Pair-level precision, recall, calibration, latency, and cost may be enough.
If positive pairs are transitively clustered
Pair-level metrics alone are not enough. You also need to inspect cluster effects and contradictions such as:
A = B
B = C
A != C
I would therefore keep the evaluation layers separate:
Blocking
- candidate recall / pair completeness
- candidates generated per record
- total candidate pairs
- blocking time
Pair verifier
- precision / recall / F1
- precision-recall curve
- calibration
- error slices
- pairs per second
Selective escalation
- percentage resolved locally
- percentage sent to the LLM
- match recall among locally resolved cases
- false matches automatically accepted
- true matches automatically rejected
- cloud cost and wait time
Clustering, if applicable
- cluster precision / recall
- over-merging
- under-merging
- largest erroneous connected components
For model validation, a random pair split can be misleading because closely related records or the same entities may appear in both train and test data. If practical, I would include one or more of:
- entity-disjoint split;
- unseen-entity split;
- new-source or new-schema split;
- chronological holdout;
- a manually reviewed production hard-negative set.
The WDC Products benchmark is useful background here because it explicitly separates corner cases, unseen entities, and different training-set sizes rather than treating every pair as identically difficult.
2. Embedding model choice: evaluate the blocker, not just semantic similarity
For short, noisy records, I would avoid selecting an embedding model solely from a general semantic-similarity leaderboard.
The embedding stage has a narrower job:
Retrieve nearly all true matches while producing few enough candidates for the expensive matcher.
That suggests evaluating each candidate generator at a controlled operating point:
For each model / representation:
1. normalize and serialize records consistently;
2. choose top-k or a similarity threshold;
3. measure candidate recall;
4. measure candidates per record;
5. measure end-to-end time through the verifier;
6. inspect which true matches were never retrieved.
A blocker that produces slightly less attractive cosine scores may still be better if it reaches the same candidate recall with half as many pairs.
I would also compare a candidate union, not only dense embedding:
dense semantic candidates
UNION lexical / BM25 candidates
UNION character n-gram candidates
UNION exact or normalized identifier rules
This is especially relevant when the decisive information is:
- a model or part number;
- one digit in a version;
- capacity, quantity, date, or unit;
- an acronym;
- a rare token;
- punctuation or spacing variation;
- an explicit contradiction.
Dense representations are useful for wording and granularity differences, while lexical and field-aware rules often preserve distinctions that semantic similarity intentionally smooths over.
Serialization is part of the model
For structured records, I would preserve the field boundary:
title: ...
brand: ...
model: ...
capacity: ...
version: ...
description: ...
rather than concatenating values without labels.
Ditto is a useful reference implementation: it serializes attributes with explicit COL and VAL markers, casts entity matching as sequence-pair classification, and includes domain-aware handling for informative spans and numbers.
It is also worth pinning the complete embedding contract:
model revision
prompt / query prefix
pooling mode
normalization
maximum sequence length
truncation policy
field serialization
dtype
Some embedding models require task-specific prefixes, and Sentence Transformers documents that prompt choice, sequence length, and truncation can change the representation. The relevant controls are described in the Sentence Transformers embedding guide.
A useful blocking comparison is therefore not:
Which model has the highest generic score?
but:
At the candidate recall I need, which representation produces
the smallest and cheapest downstream workload?
The SC-Block paper is an example of evaluating blockers at a controlled pair-completeness level and including the downstream matcher in the runtime comparison.
3. Why a task-specific verifier may help the false-positive tail
The remaining error description—“technically similar but semantically different”—sounds like a place where generic semantic similarity may be reaching its natural limit.
A bi-encoder compresses each record independently into one vector. That is excellent for scalable retrieval, but subtle contradictions can disappear in the compression.
A cross-encoder or pair classifier instead reads the two records jointly:
[record A] [SEP] [record B]
-> match / non-match
This makes token-level comparisons possible, but the training objective still matters. A generic STS or relevance reranker is trained to recognize semantic relatedness, not necessarily real-world identity. Two different products from the same series can be highly related and still be a definite non-match.
The standard route is therefore:
bi-encoder / hybrid blocker
-> task-specific pair classifier
Sentence Transformers documents this retrieve-and-rerank pattern, while Ditto is specifically designed around entity-matching pair classification.
The current errors are probably the best training data
I would prioritize:
- false positives from the current pipeline;
- pairs close to the current threshold;
- same-brand or same-family alternatives;
- variant, model-number, version, capacity, or package-size conflicts;
- local-model / LLM disagreements;
- examples from newly added sources;
- cases where the match definition itself was ambiguous.
These are more informative than a large number of easy random negatives.
A lightweight error taxonomy might be:
identifier conflict
numeric / unit conflict
version or generation conflict
variant / package-size conflict
same family, different item
different granularity
accessory versus primary item
missing decisive evidence
possible label / policy ambiguity
other
The “possible label / policy ambiguity” category matters. Some pairs cannot be resolved consistently until the application defines whether variants, bundles, revisions, parent products, or near-equivalent records count as the same entity.
Small public sanity check
In one small Walmart–Amazon control, using a fixed public split:
| Method |
Test F1 |
| Task-specific DistilBERT pair classifier |
0.800 |
| Field-aware lexical classifier |
0.718 |
| Generic STS cross-encoder |
0.382 |
| Generic bi-encoder similarity |
0.370 |
The task-specific model also made far fewer false-positive decisions on high-overlap negatives.
This is not a fair general model leaderboard:
- the task-specific and lexical models used labels;
- the semantic models were zero-shot controls;
- the dataset was product matching;
- it was not your domain;
- it did not compare against your cloud LLM;
- public entity-matching labels can contain granularity ambiguities.
The useful conclusion is narrower:
A model trained to distinguish identity from relatedness is worth testing before spending more cloud calls on the same recurring error patterns.
Sentence Transformers also has hard-negative mining utilities, but automatically mined negatives should be reviewed or filtered carefully so that true matches are not silently relabeled as negatives.
4. LLM verifier: keep it, but give it a more selective role
I would not necessarily remove the LLM. It may remain strongest for:
- a source or schema not represented in local training;
- unusual granularity differences;
- incomplete records requiring broader interpretation;
- disagreements between rules and the local verifier;
- examples close to the local decision boundary;
- provisional labeling for future training.
A possible cascade is:
if strong deterministic contradiction:
reject locally
elif local verifier is confidently positive:
accept locally
elif local verifier is confidently negative:
reject locally
else:
send to LLM / review
The thresholds should be selected on a held-out set using actual business costs, not raw softmax values.
Do not evaluate abstention using coverage alone
Suppose 95% of all pairs are easy negatives. A system can claim very high “automatic coverage” while silently rejecting many of the rare true matches.
For an abstaining system, I would report:
overall local coverage
accepted-match precision
match recall
positive-class coverage
number of true matches auto-rejected
number of false matches auto-accepted
LLM / review rate
cost per scan
risk-versus-coverage curve
The paper Confidence Calibration in Large Language Model-Based Entity Matching is relevant background: matcher scores can be overconfident, so raw confidence should not automatically be interpreted as a calibrated probability.
Structured verification
Instead of asking only for yes/no + free-form reasoning, I would make the output auditable:
{
"decision": "match | non_match | uncertain",
"supporting_fields": ["..."],
"conflicting_fields": ["..."],
"missing_decisive_fields": ["..."],
"decision_basis": "identifier | attributes | description | insufficient_evidence"
}
The free-form explanation can still be retained, but the structured fields are easier to validate, aggregate, and reuse for training.
One implementation detail: the current Ollama Structured Outputs documentation says that Ollama Cloud does not currently support schema-enforced structured outputs. If that is still true for the model/API path you use, a JSON-shaped prompt is not the same as an enforced schema. I would therefore include:
parse
-> schema validation
-> semantic validation
-> bounded retry or fallback
-> invalid-output logging
I would also avoid treating the LLM’s explanation as proof that the decision is correct. It is more useful as:
- an audit hint;
- an error-taxonomy input;
- a provisional label;
- a source of fields to check deterministically.
LLM as teacher
A practical migration path is:
1. collect difficult candidate pairs;
2. obtain LLM decisions;
3. manually audit a smaller sample;
4. train a compact student matcher;
5. retain the LLM for the student’s uncertain or shifted cases;
6. periodically retrain from reviewed production outcomes.
The recent preprint Labeling Training Data for Entity Matching Using Large Language Models studies this teacher-to-student route directly. I would treat it as promising recent evidence rather than a universal result, but it maps closely to your cost question.
The older and more operationally established analogue is active learning: the dedupe library exposes uncertain pairs for targeted labeling rather than asking for labels on random examples.
5. How I would compare the two T4s
The two-T4 question has at least five plausible layouts.
| Layout |
Description |
When it may fit |
| A |
One T4 holds both embedder and verifier |
Simple baseline; small models; scheduled batches |
| B |
Both T4s hold both models and consume a shared work queue |
Variable candidate counts; flexible load balancing |
| C |
Both T4s perform embedding, then both perform verification |
Batch-oriented pipeline; little need for stage overlap |
| D |
T4 0 embeds, T4 1 verifies |
Stable streaming pipeline with balanced stage rates |
| E |
Local embedding, cloud LLM verification |
Little local training data; cloud quality dominates cost concerns |
Why one-card co-location should be a baseline
A T4 has 16 GB of VRAM. Many BERT-family embedding models and small pair classifiers occupy only a fraction of that in inference mode.
In a representative control, MiniLM plus DistilBERT used approximately 1.1 GiB at the coarse nvidia-smi level, with substantially less live tensor allocation reported by PyTorch. That does not tell us whether your models fit, but it shows that “one model type per GPU” should not be assumed from the card count alone.
Co-location does not require simultaneous execution. Both models can remain resident while the worker alternates:
embed chunk
-> search candidates
-> verify candidate chunk
-> repeat
Why permanent stage specialization is not automatically optimal
Let:
- embedding consume
a GPU-seconds for one scan;
- local verification consume
b GPU-seconds.
In an idealized batch system where both models fit on either GPU:
- flexible use of two GPUs has a lower bound near
(a + b) / 2;
- permanent one-GPU-per-stage specialization has a lower bound near
max(a, b).
The specialized layout only reaches the flexible lower bound when the two stage loads are closely balanced. If candidate count varies or verification is much heavier, one GPU may build a queue while the other becomes idle.
Real systems add transfer, queueing, model interference, process startup, and service-isolation requirements, so this is not a performance prediction. It is only a reason not to make model type the default scheduling boundary.
What would make stage specialization reasonable?
- the two models do not fit together;
- strict streaming latency requires stage overlap;
- stage rates are stable and well balanced;
- separate services simplify reliability or deployment;
- model loading / unloading is expensive;
- one stage has an independent workload;
- isolation matters more than maximum batch throughput.
Multi-GPU embedding is still a valid option
Sentence Transformers supports passing a list of devices, or creating a reusable multi-process pool.
The same documentation notes two important details:
- multi-process overhead can be significant for smaller workloads;
- reusing the process pool is more efficient than creating it for every call.
It also distinguishes:
batch_size = records processed in one model forward batch
chunk_size = records handed to each worker process at once
Both may affect throughput.
The comparison I would run
Use the same representative scan and the same model revisions:
A. one T4, both models resident
B. two T4s, both models replicated / shared queue
C. both T4s used phase by phase
D. one T4 per stage
E. current cloud-verifier architecture
Measure:
full-scan wall time
records per second
candidate pairs per record
verified pairs per second
per-GPU utilization
queue time
peak VRAM
CPU utilization and worker count
cloud calls, tokens, cost, and wait time
quality on the same hard-negative set
A single larger GPU may win through faster compute, memory bandwidth, simpler scheduling, or lower process overhead. Two existing T4s may win on total parallel throughput or cost. Without the stage measurements and hardware prices, neither result can be assumed.
6. GPU-external bottlenecks that can hide behind low utilization
For short inputs and relatively small Transformer models, the GPU forward pass can become fast enough that fixed costs around it are no longer negligible.
Possible bottlenecks include:
source parsing
normalization
record serialization
tokenization
padding / collation
Python dispatch
kernel-launch overhead
host-to-device transfer
device synchronization
vector-index search
metadata filtering
top-k extraction
candidate-pair object construction
queueing / backpressure
JSON serialization
HTTP / TLS / network
cloud-provider queue and generation
dashboard serialization
Text Embeddings Inference uses token-based dynamic batching and exposes tokenizer workers separately from model execution.
Its CLI documentation describes max-batch-tokens as a critical hardware-utilization control and recommends increasing it until the workload becomes compute-bound. It also exposes explicit controls for tokenizer workers, concurrent requests, request batching, and backpressure.
CPU thread oversubscription
A dual-GPU deployment may run:
two model processes
x tokenizer threads
x DataLoader workers
x OpenMP / BLAS threads
If every process independently assumes it owns all CPU cores, additional workers can reduce throughput through context switching and memory contention.
This is especially plausible when the text is short: each GPU task is small, so the CPU must prepare and launch batches at a high rate.
Transfer and synchronization
The physical H2D or D2H copy may be small, while the apparent CPU time at .cpu() or .numpy() is large.
CUDA work is asynchronous. A CPU readback can become the point where the host waits for all preceding GPU work. Therefore:
large CPU time at D2H
!=
large physical copy time
The timeline and CUDA device duration should be examined together.
Candidate index and pair construction
Embedding throughput can look excellent while:
- each query is sent individually to the index;
- metadata filters prevent batching;
- embeddings move back to CPU before search;
- GPU 0 output passes through CPU before GPU 1 verification;
- Python creates millions of candidate dictionaries;
- unchanged records are repeatedly embedded;
- the ANN index is frequently rebuilt.
For Faiss specifically, the GPU documentation notes that batching queries and keeping tensors on the same GPU as the index can avoid copies and improve performance.
Queueing and external services
If cloud verification is the slowest stage, local GPUs may appear underused because the system is applying backpressure correctly.
I would distinguish:
queue wait
input preparation
model inference
output processing
network wait
rather than reporting only one “verification latency.”
The most useful optimization may then be reducing cloud-call volume, increasing safe concurrency, or changing the local/remote routing threshold—not moving the embedding model.
7. A small profiling pattern
For the PyTorch-controlled sections, Torch Profiler can collect CPU and CUDA events, tensor shapes, memory activity, and Chrome/Perfetto-compatible traces.
A minimal pattern could look like:
import torch
schedule = torch.profiler.schedule(
wait=1,
warmup=1,
active=3,
repeat=1,
)
with torch.profiler.profile(
activities=[
torch.profiler.ProfilerActivity.CPU,
torch.profiler.ProfilerActivity.CUDA,
],
schedule=schedule,
record_shapes=True,
profile_memory=True,
on_trace_ready=torch.profiler.tensorboard_trace_handler(
"./entity_match_trace"
),
) as prof:
for batch in representative_batches:
with torch.profiler.record_function("normalize_and_tokenize"):
encoded = tokenize(batch)
with torch.profiler.record_function("host_to_device"):
encoded = {
key: value.to("cuda", non_blocking=True)
for key, value in encoded.items()
}
with torch.profiler.record_function("embedding_forward"):
embeddings = embedder(**encoded)
with torch.profiler.record_function("index_search"):
candidates = search_index(embeddings)
with torch.profiler.record_function("pair_build"):
pairs = build_candidate_pairs(candidates)
with torch.profiler.record_function("verifier_forward"):
scores = verifier(pairs)
prof.step()
For sections outside PyTorch—database reads, Faiss calls, HTTP requests, JSON parsing, cloud wait—I would add ordinary monotonic timers or telemetry spans around the same named stages.
A few cautions:
- profile a short representative interval, not the entire production run;
- warm up the models first;
- compare equivalent batch shapes and token counts;
- profiling itself adds overhead;
- inspect both CPU and CUDA tables;
- use the timeline, not only aggregate rows;
- a long
.cpu() region may represent synchronization with earlier CUDA work;
- coarse
nvidia-smi utilization indicates activity, not necessarily full SM or memory-bandwidth saturation.
8. Only if candidates compete or positive pairs become clusters
This branch may be unnecessary if every result is simply an independent pair displayed on the dashboard.
If several candidates compete for one record
Independent yes/no decisions can produce:
A matches B
A matches C
but B and C are mutually exclusive
In that case, comparing candidates jointly or applying one-to-one assignment constraints may be more appropriate than isolated pair classification. ComEM is relevant background on using candidate interactions rather than treating every pair independently.
If positive pairs are transitively merged
A single false-positive edge can join two otherwise correct components. Then it becomes useful to inspect:
- inconsistent cycles;
- low-confidence bridge edges;
- oversized connected components;
- cluster-level precision and recall;
- source-specific cardinality constraints.
Graph-cleaning work such as GraLMatch is relevant in that situation.
I would not add graph cleanup merely because it exists. It becomes relevant only if the downstream system actually forms entity clusters or enforces cross-pair constraints.
A compact decision tree for the whole pipeline might be:
Is the primary problem missed true matches?
-> improve or hybridize candidate generation
Is the primary problem similar-but-different false positives?
-> train a task-specific pair verifier on production hard negatives
Does cloud verification dominate time or cost?
-> local accept/reject with selective LLM escalation
Do both local models fit one T4?
-> include co-location as the first deployment baseline
Are both T4s highly utilized and continuously supplied?
-> GPU scaling/layout comparisons are meaningful
Are the T4s idle while CPU, index, queue, or network is busy?
-> optimize the surrounding pipeline before changing GPU layout
Are accepted pairs merged into clusters?
-> add global consistency checks and cluster-level evaluation
So my provisional recommendation would be:
- keep the current high-recall candidate stage;
- build a reviewed hard-negative set from the remaining errors;
- compare the current LLM against a small task-specific local verifier;
- route only uncertain or shifted cases to the cloud;
- measure one complete scan by stage;
- treat one-T4 co-location, replicated workers, phase-wise sharing, and stage specialization as alternatives to benchmark—not architectural requirements.
The current two-T4 split may already be perfectly reasonable. I just would not infer from “two GPUs are available” that embedding is the bottleneck, that one model belongs on each card, or that more semantic similarity is the best answer to the remaining identity errors.