Project UCTF: An Open Research Program on Machine-Native AI Training Representations

Research
1

Project UCTF: An Open Research Program on Machine-Native AI Training Representations

This post is a follow-up to my earlier concept proposal on the Universal Compressed Training Format (UCTF).

The discussion that followed—especially the detailed technical feedback from @John6666—made me realize that the original concept combined several different research questions into a single proposal. Rather than trying to solve everything at once, I’ve decided to restructure the project into an open, hypothesis-driven research program.

This project is open to feedback, criticism, collaboration, and independent replication. Every stage will be documented publicly, regardless of whether the results support or challenge the original idea.

-–

Core Research Question

Can a machine-native intermediate representation serve as an alternative training representation for multilingual AI models, reducing cross-lingual semantic redundancy while preserving the information required for downstream generation?

This is a hypothesis—not a conclusion.

The purpose of Project UCTF is to investigate this question through a sequence of independent, measurable experiments.

-–

Research Philosophy

Instead of attempting to prove UCTF from the beginning, each stage will investigate one specific research question.

The goal is to let evidence shape the direction of the project.

Positive results, negative results, and unexpected findings will all be published.

If later experiments show that the hypothesis is incorrect or only partially correct, those results will still be considered valuable contributions.

-–

Why a Multi-Paper Research Program?

The original concept covered multiple independent problems:

- multilingual semantic representations

- redundancy in multilingual corpora

- machine-native representations

- language-independent training

- efficient AI training

These deserve to be investigated separately.

By splitting the work into independent papers:

- Each paper has standalone value.

- Earlier work remains useful even if later stages fail.

- Community feedback can improve each stage before moving to the next.

- The project evolves based on evidence rather than assumptions.

-–

Planned Research Roadmap

Paper 1

Measuring Semantic Redundancy in Multilingual Training Corpora

Research Question

How much semantic redundancy actually exists across multilingual training datasets?

Objective

Measure rather than assume the problem.

Possible initial evaluation resources may include multilingual benchmark datasets such as FLORES+, together with multilingual embedding baselines. The exact datasets and models will be selected after further literature review and community feedback.

Deliverables

- Redundancy measurements

- Dataset statistics

- Visualizations

- Manual inspection of representative examples

- Open notebook and reproducible analysis

-–

Paper 2

Characterizing Universal and Language-Specific Knowledge

Research Question

Which information is universal across languages, and which information is inherently language-dependent?

Objective

Develop an empirical taxonomy separating:

- universal factual knowledge

- mathematical knowledge

- scientific knowledge

- language-dependent structure

- culture-dependent expressions

- idioms

- conversational nuances

This paper defines what any machine-native representation must preserve.

-–

Paper 3

Design Requirements for a Machine-Native Intermediate Representation

Research Question

Given the findings from Papers 1 and 2, what should a UCTF-like representation contain?

Objective

Produce a formal design specification.

This stage focuses on defining requirements—not implementing them.

-–

Paper 4

Prototype Machine-Native Representation

Research Question

Can a prototype intermediate representation satisfy the requirements identified in earlier stages?

Objective

Develop an experimental prototype and evaluate:

- semantic preservation

- compression behavior

- reconstruction quality

- degradation across multiple settings

The implementation approach will be chosen based on the evidence gathered during Papers 1–3 rather than fixed in advance.

-–

Paper 5

Initial Training Validation

Research Question

Can a model trained on a machine-native representation achieve comparable downstream performance to conventional multilingual training?

Objective

Perform the first controlled comparison between:

- conventional multilingual training

- machine-native representation training

Measurements may include:

- dataset size

- representation efficiency

- training time

- computational cost

- downstream performance

- observed limitations

Regardless of the outcome, all results will be published.

-–

Open Research Principles

This project follows several principles:

- Every experiment should answer one question.

- Conclusions should follow evidence.

- Negative results are valuable.

- All code, notebooks, and intermediate findings will be released publicly whenever practical.

- Community feedback will be incorporated before major milestones.

- Credit will always be given to contributors whose feedback shapes the project.

-–

How You Can Contribute

Contributions are welcome at every stage.

Examples include:

- multilingual NLP expertise

- literature recommendations

- dataset suggestions

- methodology review

- experiment replication

- constructive criticism

- identifying prior work

- identifying flaws before implementation

Finding weaknesses early is just as valuable as proposing improvements.

-–

Current Status

The project is currently entering Paper 1.

The immediate objective is not to build UCTF, but to investigate whether multilingual semantic redundancy exists at a scale that justifies further research.

Community suggestions regarding evaluation datasets, multilingual benchmarks, encoder baselines, and experimental methodology are highly appreciated.

-–

Acknowledgment

Special thanks to @John6666 for providing detailed technical feedback on the original UCTF proposal.

Their comments significantly influenced the restructuring of this work into a staged, hypothesis-driven research program.

-–

This is an open research effort.

The goal is not to prove UCTF right.

The goal is to investigate whether the underlying hypothesis is worth pursuing.

— K7007

#UCTF #MachineLearning #NLP #MultilingualAI #OpenResearch #RepresentationLearning Research

2

For now, here are some general things that might be useful for Paper 1:

The staged roadmap makes the project much easier to reason about. For Paper 1, I would try to make the first artifact a reusable measurement map, not a proof of UCTF and not a representation design yet.

My direct suggestion would be:

Separate the measurement paths. Use clean aligned data for sanity checks, bitext/mined-bitext resources for translation-equivalent redundancy, corpus-like web data for messy training-data behavior, multiple encoders for agreement/disagreement, and a small manual audit set to calibrate what the scores actually mean.

A useful first artifact could be intentionally modest:

FLORES+ sanity check + one tiny corpus-like streaming sample + exact duplicate baseline + one lightweight multilingual encoder + threshold sweep + candidate_pairs.csv + manual_audit_template.csv.

Then add larger corpora, stronger encoders such as SONAR, and scalable dedup tools only after the audit protocol is stable.

1. A decision tree may be more useful than one more list of open questions

Instead of trying to pick one universal dataset or one universal encoder, I would split the work by what each branch is meant to answer.

If the goal is… Start with… What this answers
Pipeline sanity check FLORES+ Can the pipeline recover known cross-lingual equivalents?
Translation-equivalent redundancy OPUS-100, WikiMatrix, CCMatrix Can translation-equivalent pairs be recovered or mined?
Messy corpus-like redundancy FineWeb2, CulturaX, OSCAR, GlotCC What happens in web/pretraining-like multilingual data?
Non-semantic baseline exact duplicate count, fuzzy/MinHash near-dedup, DataTrove How much is already explained by textual duplication?
Semantic candidate mining SemDeDup, NeMo Curator semantic dedup, Sentence Transformers paraphrase mining Which pairs/clusters are meaning-similar candidates?
Encoder reliability SONAR, LaBSE, multilingual-E5, LASER/Sentence Transformers Where do encoders agree or disagree?
Low-resource analysis language-ID audit + corpus-quality audit Are the scores interpretable for low-resource slices?

The main distinction I would keep is:

Embedding similarity is a candidate generator, not the ground truth for semantic redundancy.

More detailed decision tree

A more explicit branch structure could look like this:

Branch Useful first move Notes
Controlled aligned sanity check Use FLORES+ dev/devtest for a few languages Good for checking that known translations are close
Translation-equivalent branch Use OPUS-100 / WikiMatrix / CCMatrix-like references Keep this separate from paraphrase or topic similarity
Corpus-like branch Take a tiny streaming sample from FineWeb2 / CulturaX / OSCAR / GlotCC-like data Record dataset version, filtering, dedup status, and source metadata
Exact duplicate baseline Hash normalized text Prevent exact duplicates from being counted as semantic discoveries
Near-duplicate baseline Add MinHash/fuzzy dedup later Useful, but not required for the first small notebook
Semantic candidate mining Use one lightweight multilingual encoder first Add stronger encoders after the pipeline is inspectable
Encoder agreement Compare 2–4 encoders later Disagreement is a result, not only noise
Manual audit Export candidate pairs and label match type This is what makes the numbers interpretable
Low-resource slice Check language-ID and basic text quality A low score may mean weak measurement, not low redundancy

This keeps the reply from becoming a list of demands. It gives the project author routes they can choose from.

2. Dataset roles: sanity check, bitext reference, corpus-like sample

I would avoid using one dataset for every question.

FLORES+ is useful as a controlled sanity check, but I would not use it alone to estimate training-corpus redundancy. OPUS / WikiMatrix / CCMatrix are useful for the translation-equivalent branch. FineWeb2 / CulturaX / OSCAR / GlotCC are closer to messy corpus-like conditions, but their filtering and deduplication history should be reported.

Dataset role map
Resource type Examples Good for Not enough for
Controlled aligned evaluation FLORES+ sanity check, known translation pairs, threshold calibration on clean aligned data messy training-corpus redundancy
Parallel / mined bitext OPUS-100, WikiMatrix, CCMatrix translation-equivalent redundancy paraphrase, boilerplate, same-topic-only similarity
Corpus-like multilingual web data FineWeb2, CulturaX, OSCAR, GlotCC messy pretraining-like samples raw-web claims unless processing history is included
Manual audit sample exported candidate pairs threshold calibration, false positives, label quality corpus-scale estimates by itself

FLORES+

FLORES+ is useful because it is small, multilingual, aligned, and designed for multilingual NLP / MT evaluation. I would use it for:

  • known aligned sentence pairs
  • positive-pair vs random-negative score distributions
  • checking whether encoders place translations near each other
  • threshold sanity checks

But I would keep this result separate from corpus-like redundancy.

OPUS-100 / WikiMatrix / CCMatrix

These are useful for translation-equivalent redundancy.

  • OPUS-100 is useful as an English-centric parallel-data reference. Since it is English-centric, I would not use it alone to characterize non-English ↔ non-English redundancy.
  • WikiMatrix is useful as a multilingual sentence-embedding / mined-bitext reference.
  • CCMatrix is useful as a large-scale margin-based bitext-mining reference.

I would report this branch separately from broader semantic redundancy.

FineWeb2 / CulturaX / OSCAR / GlotCC

These are better for messy corpus-like behavior.

Useful references:

FineWeb2 is especially interesting because its pipeline is documented, and the FineWeb2 repository describes per-language global deduplication and minhash_cluster_size metadata. That metadata may be useful as a non-semantic near-dedup signal.

But this also means the measurement should be framed carefully:

If using FineWeb2-like data, the measurement is on remaining redundancy after an existing filtering/deduplication pipeline, not raw web redundancy.

For corpus-like samples, I would record:

  • dataset name
  • dataset version
  • source field if available
  • language label
  • whether the data is raw / filtered / deduplicated
  • language-ID method if documented
  • existing dedup metadata if available
  • raw text
  • cleaned text
  • stable unit ID

3. Redundancy types should be separated before measuring

“Semantic redundancy” can mean several different things. I would not collapse them too early.

A small manual label set could make the measurement much easier to interpret.

Label Meaning
exact_duplicate identical or normalized-identical text
near_duplicate formatting, template, minor edits
translation_equivalent same content across languages
paraphrase_equivalent same meaning, different wording
same_fact_different_context same factual content embedded differently
boilerplate_template repeated legal text, navigation, footer, SEO template
same_topic_only same entities/topic, different claim
entity_number_date_mismatch high similarity but changed entity, number, date, relation
contradiction same entities but opposite claim
uncertain not enough context
Why this matters

High embedding similarity may mean:

  • real translation equivalence
  • paraphrase
  • boilerplate
  • same topic only
  • near duplicate
  • template reuse
  • copied structure with changed facts
  • false positive
  • contradiction

The useful artifact is not only “how many pairs are high similarity,” but what kind of high similarity they are.

Nearby references:

4. Baseline ladder: exact → near → semantic

Before embedding-based mining, I would keep the baseline ladder simple.

  1. Exact duplicate count
  2. Near duplicate / MinHash / fuzzy dedup
  3. Embedding-based semantic candidate mining
  4. Encoder agreement / disagreement
  5. Manual audit

This makes the final measurement easier to interpret.

Baseline ladder details

Level 1: exact duplicate baseline

Start with a very small baseline:

  • normalize whitespace
  • optionally lowercase where appropriate
  • hash normalized text
  • group by hash
  • count exact duplicates

This is not enough, but it prevents exact duplicates from being reported as semantic discoveries.

Level 2: near-duplicate baseline

For larger runs, fuzzy / MinHash near-dedup becomes useful.

Practical references:

The DataTrove duplicate-ID issue is useful because it shows a real audit-trail problem: if a pipeline removes documents or forms clusters, later readers need IDs and cluster membership, not only aggregate counts.

Level 3: semantic candidate mining

Only then add embedding-based mining.

References:

Level 4: encoder agreement / disagreement

If multiple encoders agree, that is a stronger candidate.

If only one encoder fires, that is still useful information.

Level 5: manual audit

Manual audit is the calibration layer. It turns “embedding neighbors” into interpretable redundancy categories.

5. Encoder baselines: useful instruments, not ground truth

I would not let one multilingual encoder define semantic redundancy.

For the first notebook, a lightweight Sentence Transformers model may be enough to validate the pipeline. SONAR can be added later as a stronger cross-lingual sentence-level baseline.

Encoder baseline details

Possible baseline set:

Encoder / family Why useful Caveat
lightweight Sentence Transformers multilingual model easy first notebook baseline not necessarily strongest cross-lingual model
multilingual-E5-small / base / large useful multilingual embedding baseline prefix / preprocessing rules matter
LaBSE classic multilingual sentence embedding / bitext retrieval baseline may not be ideal for every semantic-similarity subtype
SONAR strong cross-lingual sentence-level baseline heavier dependency; should be one instrument, not ground truth
LASER / LASER-like useful historical bitext-mining baseline check current implementation/model coverage

Useful references:

For every encoder, I would save:

  • exact model name
  • model revision if possible
  • pooling method
  • normalization method
  • input prefix / prompt rule
  • similarity function
  • language coverage notes
  • whether the unit is sentence, paragraph, or document

This is especially important for E5-style models, where prefix / prompt conventions can affect reproducibility.

6. A small first notebook could optimize for inspectability, not scale

The first notebook does not need to be a full web-scale experiment. It can be a measurement rehearsal.

A useful v0 could be:

Step Action Output
1 State purpose and non-goals readers know this is not a global estimate
2 Load FLORES+ for aligned sanity check aligned sentence table
3 Load one tiny corpus-like sample with streaming=True corpus sample table
4 Keep raw_text, cleaned_text, source, language, unit_id audit trail
5 Run exact duplicate baseline exact_duplicate_summary.csv
6 Encode with one lightweight multilingual model embedding metadata
7 Mine candidate pairs candidate_pairs.csv
8 Sweep thresholds threshold_summary.csv
9 Export manual audit template manual_audit_template.csv
10 Write observations observations.md
Notebook skeleton details

Useful implementation references:

A simple mining branch:

If sample size is… Candidate mining method
very small all-pairs cosine is transparent and easy to audit
moderate chunked semantic_search / paraphrase_mining
large FAISS / DataTrove / NeMo-style scaling later

I would not start with the biggest possible corpus. Large dedup quickly becomes an engineering benchmark: tokenization cost, RAM, I/O, distributed execution, file handles, and cluster bookkeeping.

Useful engineering reminders:

For Paper 1, I would first stabilize the measurement and audit protocol, then scale.

7. Manual audit: export candidate pairs, not only aggregate counts

The most useful v0 artifact may be candidate_pairs.csv plus manual_audit_template.csv.

Suggested columns:

Column Purpose
pair_id stable audit unit
unit_id_a, unit_id_b traceability
source_a, source_b corpus/source analysis
lang_a, lang_b language-pair matrix
raw_text_a, raw_text_b auditability
cleaned_text_a, cleaned_text_b reproducibility
encoder model comparison
score threshold analysis
threshold_band calibration
candidate_type exact / near / semantic candidate
match_label human-filled label
notes error notes
Manual audit sampling

Audit samples should include more than top-scoring pairs:

  • high-score / all-encoder agreement
  • high-score / one-encoder-only
  • near-threshold cases
  • same-script language pairs
  • cross-script language pairs
  • low-resource slices
  • suspected boilerplate/template clusters
  • same-topic false positives
  • entity/date/number mismatches
  • contradictions

This is where STS-style work is useful as background. SemEval-2017 Task 1 is not the same task as corpus redundancy measurement, but it is a useful reference for multilingual / cross-lingual semantic similarity judgments.

8. Low-resource slices need LangID and corpus-quality checks

For low-resource languages, I would be cautious.

A low redundancy score may mean:

  • actual low redundancy
  • language-ID noise
  • weak corpus quality
  • domain mismatch
  • script / orthography issue
  • closely related language confusion
  • macrolanguage vs variety confusion
  • weak encoder coverage
  • threshold calibrated on high-resource languages

So I would treat language-ID and basic corpus quality as part of Paper 1’s measurement, not just preprocessing.

Low-resource and corpus-quality references

Useful references:

  • Language ID in the Wild shows that LangID can look strong on held-out test data while failing badly on web-crawl corpora for lower-resource languages.
  • Quality at a Glance audits many language-specific corpora and reports systematic quality issues, especially in lower-resource settings.
  • GlotLID paper discusses low-resource language-ID challenges such as incorrect metadata, leakage from high-resource languages, closely related languages, macrolanguages/varieties, and noisy data.
  • GlotLID GitHub
  • GlotCC paper
  • GlotCC GitHub

A small low-resource audit could check:

  • Is the language label plausible?
  • Is the text usable natural language?
  • Is it boilerplate, OCR noise, navigation text, or code-mixed text?
  • Is it actually another closely related language?
  • Do different encoders agree?
  • Does the threshold seem calibrated for that language/script?

Even a small manual slice can prevent overinterpreting weak scores.

9. Reporting format: a redundancy map, not only one percentage

I would avoid reporting only:

“X% semantic redundancy”

A single number hides too many choices: dataset, text unit, encoder, threshold, language pair, processing history, duplicate type, and audit precision.

More reusable outputs:

Output Why useful
threshold curves shows sensitivity to cutoff
language-pair matrix shows cross-lingual structure
encoder agreement table shows measurement stability
cluster-size distribution shows whether redundancy is concentrated
dataset/source breakdown separates benchmark vs corpus-like behavior
exact/near/semantic split prevents conflating duplicate types
manual-audit label counts calibrates score interpretation
true-positive examples helps future readers understand success cases
false-positive examples helps future readers understand failure modes
false-negative examples important if using controlled aligned data
observations.md preserves qualitative lessons
How to interpret possible outcomes
Observation Possible interpretation
FLORES+ works, corpus-like data fails pipeline can recover clean aligned pairs, but messy corpus measurement is harder
exact/near duplicates dominate semantic redundancy may be smaller than expected after textual dedup
boilerplate dominates web-template redundancy should be separated from semantic content redundancy
encoders disagree strongly encoder choice is part of the measurement
low-resource scores are unstable LangID / quality / encoder coverage need audit
translation-equivalent branch works but paraphrase branch is weak later UCTF experiments should not assume all redundancy types are equally compressible

This makes Paper 1 useful even if the result is negative or mixed.

10. Minimal roadmap

One possible route:

Stage Goal Main artifact
v0 Measurement sanity notebook FLORES+ + tiny corpus sample + candidate pairs + audit CSV
v1 Add corpus realism FineWeb2 / CulturaX / OSCAR / GlotCC-like sample
v2 Add encoder comparison lightweight ST model + E5 + LaBSE + SONAR
v3 Add low-resource audit slice LangID / corpus-quality checks
v4 Scale only after audit protocol is stable DataTrove / NeMo / FAISS-style scaling

The important part is that each stage remains useful even if later stages change direction.

Why this helps later UCTF work

This would show where multilingual redundancy is:

  • exact
  • near-duplicate
  • translation-equivalent
  • paraphrastic
  • same-fact but context-dependent
  • boilerplate/template-driven
  • same-topic-only
  • encoder-dependent
  • unreliable under current measurement tools

That gives later UCTF experiments a clearer target.

If there is a strong redundancy signal, later papers can test representation bottlenecks.
If there is not, that is also useful: it means the project should narrow the claim or change the experimental target.
If the result depends heavily on language, domain, dataset, or encoder, that becomes part of the map.

3

Thank you, @John6666. This is more structured guidance than I expected, and I want to use it carefully rather than rush past it.

I’ll follow the staged approach you outlined:

v0 first — FLORES+ sanity check, one small corpus-like sample, exact duplicate baseline, one lightweight encoder, threshold sweep, and candidate_pairs.csv with manual_audit_template.csv.

I’ll separate the redundancy types before measuring rather than collapsing them into a single score, and I’ll treat embedding similarity as a candidate generator — not ground truth.

The decision tree is especially helpful. I’ll use it to keep each measurement branch answering one specific question.

I’ll publish the v0 notebook here when it’s ready for review.

Thank you for treating this as a serious research effort.

— K7007

5

Thank you @davidwarner234t. Agreed completely.

The plan for v0 follows exactly this — small auditable pipeline first, embeddings as candidate generator only, manual validation before scaling. That sequence is now the core principle of Paper 1.

— K7007

6

The staged approach is a good move. Measuring the problem first before proposing a solution will make the results much stronger. I’d pay extra attention to baselines and evaluation methods so the findings are comparable with existing multilingual NLP research. Even proving where UCTF does not work would be useful.

7

Paper 1 v0 is complete. Code, data, and results are now available on GitHub: GitHub - kesavanvjp0007/project-uctf · GitHub — feedback welcome.

8

Paper 1 — v0 Results Summary
Measuring Semantic Redundancy in Multilingual Training Corpora

The first experiment is complete. Here are the results.

Dataset

20 semantically aligned sentences across 4 languages:
English, Chinese (Simplified), Japanese, Russian
80 total sentence units.

Encoder: paraphrase-multilingual-MiniLM-L12-v2

Finding 1 — Exact Duplicate Baseline

Exact text duplicate rate: 0.0%

Text-level deduplication detects zero cross-lingual redundancy by design. The same sentence in English and Chinese shares no character overlap. This confirms that semantic redundancy measurement requires embedding-based approaches, not text matching.

Finding 2 — Cross-Language Semantic Similarity

Sample result for sentence: ‘The sun rises in the east’

English → Chinese: 0.969
English → Japanese: 0.978
English → Russian: 0.985
Chinese → Japanese: 0.984
Chinese → Russian: 0.969
Japanese → Russian: 0.985

Average cross-language similarity across all 20 sentences:

Mean: 0.9570
Minimum: 0.8529
Maximum: 0.9933

Four completely different scripts. Zero text overlap. 95.7% semantic similarity on average.

Finding 3 — Candidate Pair Mining

At similarity threshold 0.85:

Total candidate pairs found: 116
True positives (same sentence): 116
False positives (different sentence): 0
Precision: 100%

Finding 4 — Threshold Sweep

Threshold | Pairs Found | Precision
0.70 | 120 | 100.0%
0.75 | 120 | 100.0%
0.80 | 119 | 100.0%
0.85 | 116 | 100.0%
0.90 | 103 | 100.0%
0.95 | 85 | 100.0%
0.99 | 7 | 100.0%

100% precision maintained across all tested thresholds.
Zero false positives at any threshold level.

Interpretation

These results confirm three things at v0 scale:

  1. Text-level deduplication is blind to cross-lingual semantic redundancy. Standard dedup pipelines would miss all 116 semantically equivalent pairs in this dataset.

  2. Multilingual sentence encoders can reliably detect cross-lingual semantic equivalence with high precision across English, Chinese, Japanese, and Russian — including across completely different scripts and writing systems.

  3. The redundancy is substantial. At 95.7% average semantic similarity, the same information encoded across 4 languages represents significant measurable redundancy in training data.

Limitations of v0

This is a controlled small-scale experiment — 20 sentences, manually curated, perfect translations. Real training corpus behavior will differ:

  • Larger corpora will contain noisy, imperfect translations
  • Low-resource languages may show encoder degradation
  • Web-crawled data introduces domain and quality variation
  • 20 sentences is not sufficient to generalize — v1 will scale to corpus-like data

These limitations are documented. v1 will follow John6666’s recommended decision tree — adding corpus-like streaming samples, additional encoders, and a low-resource language audit slice.

Files available:

candidate_pairs.csv — 116 semantic candidate pairs
threshold_summary.csv — precision at 7 thresholds
flores_4lang.csv — aligned dataset
manual_audit_template.csv — ready for human audit
uctf_paper1_results.png — visualizations

GitHub link will be shared once the moderation hold on the previous post is cleared.

Feedback on methodology, limitations, and next steps welcome.

— K7007

9

Paper 1 — v1 Update
Real Corpus Results: 1500 Parallel Sentence Pairs

v1 is complete. Results differ significantly from v0 — which is expected and informative.

Dataset

Source: OPUS-100 parallel corpus (Helsinki-NLP)
1500 sentence pairs across 3 language pairs:
English-Chinese, English-Japanese, English-Russian
3000 total sentence units.

Encoder: paraphrase-multilingual-MiniLM-L12-v2 (same as v0)

v0 vs v1 Comparison

Metric | v0 (20 manual) | v1 (1500 real)
Mean similarity | 0.9570 | 0.7474
Min similarity | 0.8529 | -0.0447
Max similarity | 0.9933 | 1.0000
Pairs at 0.85 | 116 (100%) | 541 (36.1%)

By Language Pair

English-Chinese: mean=0.7948
English-Russian: mean=0.7886
English-Japanese: mean=0.6589

English-Japanese shows notably lower similarity than the other pairs — likely reflecting structural divergence between Japanese and English grammar rather than semantic difference.

Key Findings

  1. Real corpus data is significantly noisier than controlled translations. Mean similarity dropped from 95.7% to 74.7% — expected but important to quantify.

  2. Language pair matters. English-Japanese similarity (65.9%) is substantially lower than English-Chinese (79.5%) and English-Russian (78.9%). Compression approaches cannot assume uniform redundancy across all language pairs.

  3. Negative similarity cases exist (-0.0447). Some translation pairs diverge enough that the encoder produces near-zero or negative cosine similarity. These may represent imperfect translations, domain mismatches, or encoder limitations on structurally distant language pairs.

  4. Substantial redundancy still exists at scale. Even in noisy real corpus conditions, 67.8% of pairs exceed 0.70 similarity threshold — confirming that cross-lingual semantic redundancy is measurable and significant at corpus scale.

Limitations of v1

  • OPUS-100 is a parallel translation corpus — cleaner than real web-crawled training data
  • Only 3 language pairs tested — no low-resource languages yet
  • Single encoder used — v2 will compare multiple encoders
  • Negative similarity cases need manual inspection

GitHub

Code, data, and visualizations for both v0 and v1 now available at:

Feedback on the English-Japanese divergence and negative similarity cases especially welcome.

— K7007

10

Paper 1 v2 complete. Compared 4 encoders (MiniLM, mE5, LaBSE, mpnet) on 1500 real parallel pairs using GPU. mE5-small emerged as the strongest baseline — mean similarity 0.8820, std deviation 0.0455, zero negative similarity cases, and near-elimination of the English-Japanese divergence gap observed in v1.

11

mE5-small demonstrated strong robustness to three types of realistic training data noise — character-level errors (96.43% mean similarity to clean), word-order shuffling (97.50%), and code-switching (94.87%). All noise types maintained above 94% similarity to clean text, with 98.7–100% of pairs exceeding the 0.85 threshold. This suggests mE5 is a viable encoder for real-world noisy multilingual training corpora where perfect translations cannot be assumed.

12

mE5-small maintained strong semantic similarity detection for low-resource languages — Sinhala (0.9083), Nepali (0.9040), and Kyrgyz (0.9013) — performing comparably to high-resource language pairs like English-Chinese (0.8694) and English-Japanese (0.8630). This suggests mE5 may be a viable encoder baseline for UCTF even in low-resource settings, though the higher standard deviation for Kyrgyz (0.0582 vs 0.0410 for Sinhala) indicates increased variability that warrants further investigation.

13

At corpus scale (10,000 pairs, 20,000 sentences, 5 languages), mE5-small encoded 20,000 sentences in 13.2 seconds (1,515 sentences/second) using only 458 MB VRAM on a consumer RTX 3050 GPU — demonstrating practical feasibility for resource-constrained environments. Critically, low-resource languages (Sinhala: 91.84%, Nepali: 90.41%) outperformed high-resource languages (Chinese: 87.55%, Japanese: 86.67%) at corpus scale, a counterintuitive finding that warrants further investigation in Paper 2.

14

Project UCTF — Paper 1 Complete Results
Measuring Semantic Redundancy in Multilingual Training Corpora

Paper 1 is complete..

Full PDF report and all code, data, and notebooks are available on GitHub:

What we measured

We tested whether cross-lingual semantic redundancy in multilingual AI training data is measurable, substantial, and encoder-dependent — and whether an embedding-based approach can detect it reliably.

Experiment overview

Version | Scale | Setup | Mean similarity
v0 | 20 pairs | Manual aligned, 4 languages, MiniLM | 0.9570
v1 | 1,500 pairs | OPUS-100 real corpus, MiniLM | 0.7474
v2 | 1,500 pairs | 4 encoders compared on GPU | 0.8820 (mE5)
v3a | 900 sentences | 3 noise types, mE5 | 0.9627 avg
v3b | 600 pairs | Nepali, Sinhala, Kyrgyz, mE5 | 0.9045 avg
v4 | 10,000 pairs | 5 languages, corpus scale, GPU | 0.8948

Total pairs analyzed: 13,320
Languages tested: English, Chinese, Japanese, Russian, Nepali, Sinhala, Kyrgyz
Hardware: NVIDIA RTX 3050 4GB (consumer laptop GPU)

Finding 1 — Text deduplication fails completely

Exact duplicate detection rate: 0.0%

Standard text-level deduplication finds zero cross-lingual redundancy by design. The same sentence in English and Chinese shares no character overlap. This confirms that semantic redundancy measurement requires embedding-based approaches — not text matching.

Finding 2 — mE5-small is the strongest encoder baseline

Encoder comparison at 1,500 pairs (v2):

Encoder | Eng-Chi | Eng-Jap | Eng-Rus | Overall | Neg. pairs | Std dev
MiniLM | 0.7948 | 0.6589 | 0.7886 | 0.7474 | 1 | 0.1878
mE5 | 0.8694 | 0.8630 | 0.9136 | 0.8820 | 0 | 0.0455
LaBSE | 0.7898 | 0.6149 | 0.7855 | 0.7301 | 0 | 0.1990
mpnet | 0.8249 | 0.7044 | 0.8176 | 0.7823 | 0 | 0.1733

mE5-small outperforms all tested encoders on three criteria simultaneously: highest mean similarity (0.8820), lowest standard deviation (0.0455), and zero negative similarity pairs. It also reduces the English-Japanese divergence gap from 0.136 (MiniLM) to 0.006 — a finding that directly addresses the low-resource language concern raised in earlier feedback.

Finding 3 — mE5 is robust to real-world noise (v3a)

Noise type | Mean sim to clean | Above 0.85 threshold
Word shuffle (30% swapped) | 0.9750 | 100.0%
Char noise (5% OCR errors) | 0.9643 | 100.0%
Code switching (20% foreign) | 0.9487 | 98.7%

mE5 maintains 94.9–97.5% similarity to clean text under three types of realistic noise. This suggests the encoder is viable for real training corpora where perfect translations cannot be assumed.

Finding 4 — Low resource languages match high resource (v3b + v4)

At corpus scale (2,000 pairs each):

Language | Type | Mean similarity | Above 0.85
Sinhala | Low resource | 0.9184 | 93.5%
Russian | High resource | 0.9092 | 91.3%
Nepali | Low resource | 0.9041 | 89.3%
Chinese | High resource | 0.8755 | 77.6%
Japanese | High resource | 0.8667 | 67.2%

Sinhala (low resource) outperforms Chinese and Japanese (high resource) at corpus scale. This is counterintuitive and we do not have a confident explanation yet. It is flagged as a priority investigation for Paper 2.

Finding 5 — Consumer hardware is sufficient

20,000 sentences encoded in 13.2 seconds
1,515 sentences per second
458 MB VRAM used out of 4,096 MB available
Hardware: NVIDIA RTX 3050 Laptop GPU

Corpus-scale multilingual semantic redundancy measurement is practical on consumer hardware. This is relevant for the UCTF democratisation goal.

Limitations

OPUS-100 is a parallel translation corpus — cleaner than real web-crawled training data. Only 7 languages tested — broader coverage needed. Noise simulation is controlled — real corpus noise is more varied. No manual audit of candidate pairs completed yet. Negative similarity cases require manual inspection.

Conclusion

Cross-lingual semantic redundancy is measurable, substantial, and encoder-dependent. mE5-small is a viable baseline for a UCTF compression layer — robust to noise, effective across language families including low-resource settings, and practical on consumer hardware.

The core UCTF hypothesis — that a machine-native compressed representation could reduce multilingual training data redundancy — is supported by these preliminary findings.

What comes next — Paper 2

Characterizing which information is universal vs language-specific.
Manual audit of candidate pairs.
SOSAR and LaBSE comparison at corpus scale.
Test on web-crawled corpora (FineWeb2, CulturaX).
Investigate the low-resource language outperformance finding.

Resources

Full PDF report: github.com/kesavanvjp0007/project-uctf/paper1_final
All code and notebooks: github.com/kesavanvjp0007/project-uctf/paper1_v0 through v4
Candidate pairs CSV: 116 semantic pairs with similarity scores
Threshold sweep CSV: precision at 7 thresholds
Encoder comparison CSV: 6,000 records across 4 encoders

Feedback on the low-resource outperformance finding and the noise robustness results especially welcome.

Special thanks to @John6666 for the research architecture guidance that shaped this entire program.

15

Hi. Um, taking everything so far into account, maybe something like this:

This has moved well beyond the original v0 sanity check. The v0–v4 sequence now looks like a substantial exploratory measurement package with three useful branches:

  1. translation-equivalent sentence measurement,
  2. multilingual encoder comparison,
  3. exploratory noise and low-resource observations.

My short reading is:

The experiments provide useful evidence that translation-equivalent sentences can be detected in multilingual embedding spaces, and that the measured result depends materially on the encoder and dataset setup.

I would not rush to add many more large runs immediately. The least disruptive route may be:

  1. make the existing package easy to inspect,
  2. add one small calibration slice if useful,
  3. then let the work branch naturally toward Paper 2, a web-corpus extension, or later representation-preservation tests.

None of those branches needs to become an obligation or a blocker for the others.

Where the work seems to be now

Branch Earlier stage Current result Low-cost next option
Translation alignment FLORES+ controlled sanity check OPUS-based aligned-pair measurements at larger scale compare positives with mismatched or hard-negative pairs
Encoder comparison one lightweight baseline four encoders compared; mE5 performed best in this setup preserve the exact preprocessing and optionally add ranking or margin diagnostics
Noise proposed failure categories character noise, code switching, and word shuffle tested separate surface-noise invariance from sensitivity to meaning changes
Low-resource slice initially an uncertainty Sinhala and Nepali produced unexpectedly high scores inspect per-language negative separation, sampling stability, and high/low tails
Training-corpus generalization planned branch experiments remain primarily parallel-corpus based optional small FineWeb2/CulturaX-like branch
Reproducibility public release planned results and resource links reported in the thread one stable Paper 1 page linking the artifacts and experiment settings
Representation/compression later-paper question similarity evidence now exists preservation, reconstruction, compression, and training tests later
Paper 2 taxonomy planned the low-resource result creates an additional question conceptual work can begin in parallel

Directly on the two feedback questions

Low-resource result: I would not dismiss the Sinhala/Nepali result, but resource level alone probably does not explain it. It may be a real property of this particular dataset–encoder–sampling combination. The most informative lightweight check would probably be to compare the positive and mismatched-negative score distributions separately for each language.

Noise result: character corruption and limited code switching look like useful invariance tests. Word-shuffle similarity is more ambiguous: it may indicate robustness, but it may also indicate that cosine similarity is not very sensitive to word order. That makes it an interesting diagnostic result rather than necessarily a direct simulation of real-corpus robustness.

One practical note: when I checked the public Project UCTF repository, the public view still showed only the short README, and the report/notebook paths linked in the thread did not resolve from that view. That may simply be upload timing, propagation, or an incomplete public view. Even one stable Paper 1 README linking the report, notebooks, CSVs, environment, and limitations would make the package much easier to inspect.

What the current experiments appear to establish

I would separate the direct observations from the later hypotheses roughly like this:

Reported observation Relatively direct reading Question left for another branch
Exact duplicate rate was 0% across translated pairs exact text matching does not recover differently written cross-lingual translation pairs how much is detected by fuzzy, semantic, or multilingual-mining methods
Aligned OPUS pairs received high similarity tested encoders frequently place known translations close together how often arbitrary web-corpus neighbors are actually redundant
Encoder results differed substantially encoder choice is part of the measurement which encoder is preferable for clustering, mining, reconstruction, or training
mE5 produced the strongest scores in v2 mE5 performed best among the four tested encoders in this OPUS-based setup whether it is also best for candidate discrimination or representation preservation
Synthetic perturbations retained high similarity the tested embedding was stable under those transformations whether it distinguishes meaning-preserving changes from meaning-changing changes
Sinhala/Nepali scores were high those OPUS slices aligned strongly under mE5 whether this comes from language, corpus composition, translation style, sampling, or model calibration
20,000 sentences were embedded quickly on an RTX 3050 embedding generation at that tested scale was feasible on the reported laptop GPU full mining, clustering, decoding, compression, and training cost
Cross-lingual similarity is measurable a shared embedding space is a useful measurement instrument whether redundant content can be merged or compressed without losing required information

This is already a useful result. It supports moving from “does a measurable signal exist?” toward “what exactly does that signal represent?”

I would keep the original paper boundaries because they form a useful separation of questions:

  • Paper 1: measure redundancy candidates and their structure,
  • Paper 2: distinguish potentially universal and language-specific information,
  • Paper 3: define representation requirements,
  • Paper 4: test prototype preservation, reconstruction, and compression,
  • Paper 5: test downstream training behavior.

Under that division, Paper 1 does not need to carry the full compression-layer claim. It can instead provide a more precise result:

Translation-equivalent similarity is measurable under the tested conditions, but its interpretation depends on encoder choice, dataset structure, negative separation, and human audit.

How I would read the low-resource result

The Sinhala/Nepali result is interesting because it runs against a simple expectation that higher-resource languages must always align better. Existing multilingual-retrieval research suggests that low-resource behavior is not determined by “resource level” alone.

There are examples of strong transfer to low-resource bitext mining, including OneAligner and earlier LASER-based filtering for Nepali–English and Sinhala–English. There are also examples where zero-shot alignment for a lower-resource language is weak until some adaptation is added. The practical pattern seems to be that model training, language pair, domain, and corpus construction all matter.

So my first interpretation would be:

The result may be real for these OPUS slices and this encoder, but “low-resource” is probably not the causal variable by itself.

OPUS-100 is English-centric: its training pairs include English on either the source or target side. Its language subsets may also differ in source corpus, sentence length, translation literalness, domain, formatting, and alignment quality. Any of those could affect the mean score.

A low-effort diagnostic tree could be:

If this happens… A reasonable interpretation
the language ordering remains similar across several random samples probably a stable dataset/model interaction
the ordering changes strongly across seeds sampling composition is important
Sinhala/Nepali remain high for all tested encoders corpus or alignment characteristics may be important
the result is mainly present for mE5 encoder-specific calibration or training coverage may be important
positive scores are high but mismatched negatives are also high absolute cosine is high, but discrimination may not be
high scores are concentrated in short or formulaic sentences corpus composition may explain part of the effect
inspection of the high/low tails finds alignment differences translation or data-quality differences may be contributing
the result persists for non-English↔non-English pairs it is less likely to be only an English-pivot effect

If only one check is convenient, I would choose this:

For each language, compare the known translation-pair distribution against a same-size mismatched-pair distribution.

That gives more information than the positive mean alone while remaining small enough to run with the current setup.

If the result survives resampling and negative comparison, it becomes a useful dataset–encoder interaction to carry into Paper 2. If it changes substantially by encoder or sample, that is also informative: the result can be reported as calibration or corpus sensitivity rather than a general low-resource advantage.

For a later web-corpus branch, low-resource interpretation becomes more difficult because language identification and corpus quality become part of the measurement. Language ID in the Wild found severe LangID failures for many lower-resource web-crawl corpora despite strong held-out test results, and Quality at a Glance found systematic quality and labeling problems across lower-resource web corpora.

That does not make low-resource analysis impractical; it means language-ID and basic corpus-quality checks should remain visible in the measurement.

How I would read the noise results

The three perturbations appear to test somewhat different properties.

Perturbation What high similarity may indicate What it does not establish by itself
character corruption stability to limited surface/OCR-like damage robustness to arbitrary extraction or encoding failures
limited code switching stability when some lexical material changes language preservation under natural code switching with semantic shifts
word shuffle stability to token-order changes preservation of syntax or proposition-level meaning

The word-shuffle result is especially interesting. A sentence encoder can remain close under shuffling because it is robust to noisy ordering, but it can also remain close because its cosine geometry is driven heavily by lexical content.

SenTest studies sentence-encoder behavior under perturbations including word-order changes. ALIGN-SIM evaluates sentence embeddings using semantic distinction, synonym replacement, antonym replacement, paraphrasing, and sentence jumbling. These provide useful ways to separate invariance from semantic sensitivity.

A practical decision tree could be:

  • If the goal is robustness to surface corruption: keep character noise and meaning-preserving code switching.
  • If the goal is sensitivity to meaning: add controlled entity, number, date, antonym, and negation changes.
  • If the goal is realistic web noise: sample actual boilerplate, truncation, HTML extraction errors, OCR errors, repeated navigation text, and bad alignments.
  • If shuffled sentences remain very close: report this as a word-order-sensitivity observation, not only as robustness.

A compact control set could reuse the same base sentences:

Pair construction Desired relative behavior
faithful translation high similarity
faithful paraphrase high similarity
synonym replacement high similarity
limited character corruption moderately high similarity
meaning-preserving code switching high similarity
changed entity lower similarity
changed number or date lower similarity
antonym replacement lower similarity
negation reversal lower similarity
random mismatched translation low similarity
shuffled text whose meaning becomes unclear or changes lower similarity if order-sensitive meaning is captured

Negation is worth checking because embedding-based metrics can sometimes assign high similarity to statements whose truth conditions have been reversed. Negation-aware Evaluation of Language Generation Systems gives examples of this failure mode.

This does not need to become a large benchmark. A few deliberately constructed pairs can clarify what the current “noise robustness” score is preserving.

One small calibration layer, if useful

If one modest addition were chosen before moving on, I would probably favor a compact calibration slice rather than another large positive-pair run.

The exact size is flexible. Even something on the order of 100–200 deliberately selected pairs could connect the similarity distributions to interpretable categories.

Possible sampling:

  • highest-scoring true translation pairs,
  • lowest-scoring true translation pairs,
  • pairs near the current threshold,
  • random mismatched pairs,
  • nearest non-matching candidates,
  • same-topic but different-claim pairs,
  • changed entity/date/number pairs,
  • negation or contradiction pairs,
  • cases where encoders disagree,
  • selected high/low tails from Sinhala and Nepali.

Possible labels:

  • translation_equivalent
  • paraphrase_equivalent
  • near_duplicate
  • same_fact_different_context
  • boilerplate_template
  • same_topic_only
  • entity_number_date_mismatch
  • contradiction
  • bad_alignment
  • uncertain

The useful output would not need to be complicated:

Output What it adds
precision by threshold band helps interpret the threshold sweep
positive vs mismatched-negative distributions shows discrimination
false-positive categories shows what the embedding confuses
false-negative examples shows what translations it misses
encoder-disagreement examples helps compare the baselines
short annotation notes preserves how ambiguous cases were handled

A low-burden version could be selected according to the current interest:

  • If the low-resource result is the focus: inspect Sinhala/Nepali high and low tails.
  • If mE5 is the focus: inspect examples where mE5 disagrees with the other encoders.
  • If thresholding is the focus: inspect around the threshold and the highest-scoring mismatches.
  • If Paper 2 is the focus: label whether each candidate preserves universal content, language-specific expression, or both.

This reconnects the current work with the v0 principle that embedding similarity is a candidate generator rather than ground truth.

Alignment score versus candidate-mining quality

Mean positive-pair cosine answers one useful question:

Are known aligned translations placed close together?

It does not fully answer another question:

Can the system distinguish the correct counterpart from nearby incorrect candidates?

For that second question, useful diagnostics include:

Diagnostic Question answered
positive mean cosine are known pairs aligned?
positive–negative score gap are correct pairs separated from mismatches?
Recall@1 is the correct translation ranked first?
MRR how high is the correct translation ranked?
local margin score is the pair unusually close relative to its neighborhoods?
encoder agreement is the candidate stable across measurement instruments?

The distinction matters because absolute cosine scales can differ by model, language, and local embedding neighborhood. Margin-based Parallel Corpus Mining was proposed because a single hard cosine threshold can suffer from scale inconsistency; it compares a candidate pair with nearby alternatives instead.

A simple branch would be:

  • If the goal is alignment measurement: the current positive-pair means remain useful.
  • If the goal is candidate mining: include mismatched candidates and report retrieval or margin results.
  • If the goal is dedup thresholding: calibrate threshold behavior by encoder and possibly by language pair.
  • If the goal is representation design: move beyond similarity to preservation and reconstruction tests.

This slightly changes how I would describe the mE5 result:

mE5 produced the strongest positive-pair alignment in the current comparison.

If its negative separation and retrieval ranking are also strongest, the broader baseline conclusion becomes stronger naturally.

mE5 is a plausible strong result here because the E5 family is trained contrastively on large multilingual text-pair collections and uses hard-negative training. The multilingual-e5-small model card also documents an important reproduction detail: for symmetric tasks such as semantic similarity, bitext mining, and paraphrase retrieval, both inputs should use the query: prefix.

It may therefore help to preserve:

  • exact model identifier and revision,
  • prefixes or prompts,
  • pooling,
  • embedding normalization,
  • truncation length,
  • score function,
  • sampling seed.
Translation-pair branch versus training-corpus-like branch

The current OPUS results appear most directly useful as a translation-equivalent branch of Paper 1.

That is already valuable. It establishes a cleaner, controlled form of cross-lingual semantic overlap at larger scale than v0.

A separate branch is needed only if the desired statement becomes:

What proportion and type of semantic redundancy remain in multilingual pretraining corpora?

Those corpora contain additional phenomena:

  • exact and near duplicates,
  • translated pages,
  • paraphrases,
  • common factual descriptions,
  • repeated templates,
  • cookie/legal/navigation boilerplate,
  • OCR or extraction damage,
  • language-ID errors,
  • same-topic but non-equivalent text,
  • copied text with changed dates, entities, or numbers.

A decision tree could be:

  • If the goal is to finish the translation-equivalent branch: add negatives, ranking/margin evaluation, or a small audit.
  • If the goal is broader training-corpus claims: add one small web-corpus sample.
  • If the goal is representation compression: move to preservation/reconstruction rather than enlarging Paper 1 indefinitely.
  • If Paper 2 is already the priority: begin it while leaving the web-corpus branch explicitly open.

FineWeb2 may be a practical corpus branch because it exposes language, script, language score, Common Crawl metadata, and minhash_cluster_size. Its repository explains that the corpus was deduplicated globally per language and that the retained document stores the duplicate-cluster size.

That creates a useful comparison:

Signal Possible interpretation
minhash_cluster_size existing textual/near-duplicate structure
semantic nearest neighbors possible meaning-level structure
boilerplate label template repetition
language/source metadata distribution and data-quality context
manual label actual relation between examples

The interpretation should remain:

remaining redundancy after FineWeb2’s existing pipeline,

rather than raw-web redundancy.

A small streaming sample would be enough for a first comparison; it would not need to become a web-scale dedup project immediately.

Making the current package easier to inspect

The highest-value release artifact may be one stable Paper 1 research card, rather than a complicated repository structure.

Something like:

What was tested

  • v0–v4 experiment summary,
  • datasets and subsets,
  • languages and pair counts,
  • encoders,
  • perturbations,
  • metrics,
  • hardware.

What was not tested

  • large noisy web corpora,
  • human precision of semantic candidates,
  • representation reconstruction,
  • information-loss curves,
  • downstream training,
  • full pipeline compute cost.

Reproduction information

  • dataset revision/subset/split,
  • sampling procedure and random seed,
  • exact encoder revision,
  • prefixes and preprocessing,
  • pooling and normalization,
  • thresholds,
  • environment or requirements,
  • notebook execution order.

Linked artifacts

  • report,
  • notebooks or scripts,
  • result CSVs,
  • candidate pairs,
  • hardware log,
  • known limitations,
  • open follow-up branches.

The Hugging Face dataset-card guide uses a similar general structure: keep the artifact context, language, license, size, intended use, and limitations close to the artifact. A lightweight Paper 1 card can apply the same idea without requiring a large documentation effort.

At the time of this check, the Project UCTF repository still displayed one commit and the short README in its public tree. Since forum and repository updates can appear out of order, this may simply be a timing or visibility issue. The main practical point is to have one stable landing page once the files are visible.

Preserving stable IDs is also useful. A DataTrove MinHash discussion shows the difficulty of reconstructing duplicate membership from removal and cluster files after the fact. Keeping pair IDs, source IDs, scores, and cluster IDs early avoids that problem later.

Setup-qualified wording

The findings are useful. Setup-qualified wording can keep each conclusion attached to the experiment that produced it.

Broad summary Setup-qualified version
Text deduplication fails completely Exact text matching does not recover the tested translation-equivalent cross-lingual pairs
Semantic redundancy is substantial in multilingual training data High translation-equivalent similarity was observed in the tested parallel-corpus settings
mE5 is the strongest encoder baseline mE5 performed best among the four tested encoders in this OPUS-based comparison
mE5 is robust to real-world noise mE5 remained stable under the three tested synthetic perturbations
Low-resource languages match or outperform high-resource languages Sinhala and Nepali produced unexpectedly high scores in the tested OPUS slices
Consumer hardware is sufficient embedding generation for 20,000 tested sentences was feasible on the reported RTX 3050 setup
mE5 is viable as a UCTF compression layer mE5 is a promising measurement encoder and later representation candidate
The core UCTF hypothesis is supported the findings strengthen the motivation for the next UCTF experiments

This is not about reducing the result. It keeps measurement, representation, and training claims in their respective experimental branches.

The encoder qualification is especially useful because MTEB found that no single embedding method dominated across all evaluated tasks. MMTEB has since expanded multilingual embedding evaluation to more than 500 tasks across 250+ languages, further emphasizing task, language, and domain dependence.

Possible parallel routes from here

I do not think every Paper 1 extension has to block Paper 2.

A possible branch map:

Current priority Natural route
preserve the current research record publish one stable Paper 1 card and artifact links
understand the low-resource result per-language negatives, resampling, and tail inspection
strengthen mE5 baseline selection ranking/margin evaluation and encoder disagreement analysis
understand noise behavior meaning-preserving vs meaning-changing controls
support broader training-corpus claims small FineWeb2/CulturaX-like branch
begin Paper 2 start the universal vs language-specific taxonomy in parallel
test UCTF representation viability preserve for later reconstruction, bottleneck, and downstream-training experiments

A default route with minimal disruption could be:

  1. publish or stabilize the current Paper 1 artifact,
  2. select one small calibration slice,
  3. begin Paper 2 conceptually,
  4. leave web-corpus and representation branches explicitly open.

If the low-resource result becomes central to Paper 2, the smallest useful bridge would be:

verify that the result survives mismatched negatives and resampling, then treat it as a dataset–encoder interaction to explain rather than a general low-resource advantage.

For the broader ShinBay-UCTF framework, I think the present results are enough to motivate using a shared semantic space as a provisional Layer-1 interface.

I would keep “validated compressed representation” for later tests of:

  • information preservation,
  • reconstruction,
  • controlled forgetting or merging,
  • provenance,
  • deletion semantics,
  • downstream generation or training behavior.

That keeps the connection between the two threads clear:

Paper 1 supplies evidence and measurement tools; later papers determine what the architecture can safely do with the representation.

References directly relevant to this checkpoint

Current project

Parallel corpus and candidate mining

Sentence-embedding interpretation

Optional web-corpus branch

Overall, I think the current work has already produced a useful Paper 1 checkpoint.

The most informative next move does not have to be larger. It may simply be to preserve the current setup, add one small discrimination or calibration layer, and let the next branch be selected by the question:

  • taxonomy: Paper 2,
  • training-corpus prevalence: web-corpus branch,
  • encoder quality: negatives, ranking, or margin,
  • representation viability: preservation and reconstruction later.

That would keep the program moving without forcing every open branch to be resolved at once.

16

@John6666 — README has been updated with full navigation to all artifacts. All CSVs, notebooks, and PDF reports are now linked directly from the README. Your feedback on word-shuffle has been added to the limitations section. Paper 2 taxonomy results are posted separately in this thread. GitHub - kesavanvjp0007/project-uctf · GitHub — K7007

17

Project UCTF — Papers 1 & 2 Complete
Measuring Redundancy · Characterizing Knowledge · Defining What UCTF Can Compress

Two papers complete. Here is the combined results report.

All code, data, notebooks, and PDF reports are available on GitHub:

Paper 1 — Measuring Semantic Redundancy in Multilingual Training Corpora

Research question: How much semantic duplication exists across languages in real AI training datasets?

Six experiments — v0 through v4:

Version | Scale | Mean similarity | Key finding
v0 | 20 pairs manual | 0.9570 | 100% precision at all thresholds
v1 | 1,500 real pairs | 0.7474 | Real data noisier — Eng-Jap gap of 0.136
v2 | 4 encoders GPU | 0.8820 mE5 | mE5 best — zero negatives, std 0.046
v3a | 900 noisy | 0.9627 avg | mE5 robust — 98.7–100% above 0.85
v3b | 600 low-resource | 0.9045 avg | Sinhala, Nepali match high-resource
v4 | 10,000 corpus | 0.8948 | 1,515 sent/sec on RTX 3050, 458 MB VRAM

Paper 1 findings:

  1. Text deduplication finds 0% cross-lingual redundancy. Embedding-based approaches are necessary.
  2. mE5-small is the strongest encoder — mean 0.8820, std 0.0455, zero negative pairs, closes Eng-Jap gap from 0.136 to 0.006.
  3. mE5 maintains 94.9–97.5% similarity under character noise, word shuffling, and code-switching.
  4. Low-resource languages (Sinhala 91.8%, Nepali 90.4%) outperform high-resource languages (Chinese 87.6%, Japanese 86.7%) at corpus scale — reason under investigation.
  5. Consumer hardware is sufficient — 20,000 sentences in 13.2 seconds on RTX 3050 laptop GPU.

Paper 2 — Characterizing Universal and Language-Specific Knowledge

Research question: Which semantic information is universal and safely compressible — and which is language-dependent and must be preserved?

Dataset: 60 sentences × 6 knowledge categories × 4 languages = 240 sentences

Categories: factual · mathematical · named_entity · emotional_nuance · cultural_idiom · grammar_dependent

Results — similarity by category (mE5-small):

Category | Mean | Std | Above 0.90 | UCTF Safety
grammar_dependent | 0.9148 | 0.0213 | 76.7% | Safe
emotional_nuance | 0.9069 | 0.0244 | 60.0% | Safe
factual | 0.8968 | 0.0216 | 46.7% | Safe
mathematical | 0.8795 | 0.0309 | 25.0% | Medium
named_entity | 0.8749 | 0.0318 | 16.7% | Medium
cultural_idiom | 0.8573 | 0.0470 | 20.0% | Caution

Most universal sentence: “She gave him a gift yesterday.” — sim=0.9459 (grammar_dependent)
Most divergent sentence: “It is raining cats and dogs.” — sim=0.8103 (cultural_idiom)

Paper 2 findings:

  1. Hypothesis partially reversed. Grammar-dependent sentences scored highest (91.5% preserved) — not factual or mathematical. mE5 abstracts syntactic structure and captures semantic intent regardless of word order.

  2. Cultural idioms are the primary compression risk. Lowest preservation (85.7%), highest variance (std 0.0470), only 58.3% of pairs above 0.85 threshold. Idiomatic expressions carry language-specific meaning that compression would destroy.

  3. Emotions are more universal than expected. Emotional nuance scored 90.7% — human emotional concepts transfer across language boundaries more reliably than initially assumed.

  4. Named entities show moderate divergence (87.5%) with high variance. Cultural references (Mona Lisa, Great Wall) diverge more than scientific proper nouns (Einstein, Amazon River).

  5. Hypothesis partially reversed — grammar and emotion outperformed factual and mathematical categories, suggesting mE5 measures semantic intent rather than syntactic universality.

UCTF Design Implication — 3-Tier Compression Strategy

Based on Papers 1 and 2, the UCTF encoder requires a tiered approach:

Tier 1 — Safe compression
Categories: factual, mathematical, grammar-dependent, emotional
Preservation: 88–91%
Strategy: Full UCTF compression — low risk of information loss

Tier 2 — Flagged compression
Categories: named entities
Preservation: 87.5%
Strategy: Compress with entity flagging — preserve proper nouns separately

Tier 3 — Preserve as-is
Categories: cultural idioms
Preservation: 85.7%
Strategy: Store as language-specific tokens with source language tag — do not compress

Information Preservation Estimates

grammar dependent → 91.5% preserved
emotional nuance → 90.7% preserved
factual → 89.7% preserved
mathematical → 88.0% preserved
named entity → 87.5% preserved
cultural idiom → 85.7% preserved

Limitations

Paper 1: OPUS-100 is a parallel corpus — cleaner than web-crawled training data. Only 7 languages tested.
Paper 2: 10 sentences per category is a small sample. Only 4 languages. mE5-small used — larger encoders may differ.
Both: Preservation estimates are proxy measures based on similarity scores — actual UCTF information loss requires encoder-decoder experiments in Paper 4.

What Comes Next — Paper 3

Formal design specification for the UCTF encoder.
Tiered compression architecture based on Papers 1 and 2 findings.
UCTF token format specification.
Encoder requirements per compression tier.
SOSAR evaluation as candidate encoder for Tier 1.

Resources

GitHub: GitHub - kesavanvjp0007/project-uctf · GitHub
Paper 1 PDF: /paper1_final/
Paper 2 PDF: /paper2/
All code, datasets, embeddings, and visualizations: /paper1_v0 through /paper2/

Feedback on the cultural idiom finding and the 3-tier compression strategy especially welcome.

Special thanks to @John6666 for the research architecture that made this program possible.

— K7007