Diffsingers What's the best way to make A word to phoneme .Lab converter for Vlabeler?

The extension alone does not define the data contract.

For example, MakeDiffSinger’s acoustic forced-alignment workflow uses a WAV and a corresponding .lab transcription as input to an alignment process. In SOFA’s inference workflow, a WAV and a same-named .lab transcription are also paired; the transcription is converted to phonemes before alignment.

That is different from an HTK/Sinsy-style timed label such as:

0 5000000 sil
5000000 6300000 k
6300000 9100000 a

It is also different from an NNSVS full-context label, a Praat TextGrid with several tiers, or DiffSinger metadata fields such as phoneme sequences and durations.

Before implementing the model, I would freeze one “golden fixture” containing:

input text
input audio, if applicable
expected phoneme sequence
expected phoneme timing
expected final file
time unit
boundary convention
special-symbol convention

For example:

text:
hello

canonical phones:
HH AH L OW

audio:
hello.wav

expected internal timing:
[
  {"phone": "HH", "start": 0.12, "end": 0.20},
  {"phone": "AH", "start": 0.20, "end": 0.36},
  ...
]

expected export:
hello.TextGrid
or
hello.lab

Once this fixture exists, it becomes possible to test the format writer independently from the G2P and aligner.

TextGrid is often a convenient internal interchange format because it can preserve word, phoneme and other annotation tiers and can be inspected in Praat or vLabeler. vLabeler is explicitly designed around customizable labelers, plugins and scripts; its repository documents custom labelers and scripting/plugin support. A TextGrid-oriented implementation is also available in vlabeler-textgrid.

However, the capabilities of a particular TextGrid importer still need to be checked. For example, an importer may support interval tiers but not every Praat object or point tier. A round-trip test is therefore useful:

model output
→ import into vLabeler
→ edit/save
→ reopen or convert back
→ compare phone count, labels and boundaries

I would treat the system as a set of explicit contracts rather than one opaque model:

raw text
  ↓
normalization/tokenization
  ↓
pronunciation candidates
  ↓
canonical phoneme sequence
  ↓
lyrics–performance matching
  ↓
coarse word/syllable alignment
  ↓
fine phoneme alignment
  ↓
format conversion
  ↓
vLabeler inspection/correction

Text normalization

This includes decisions about:

• capitalization;
• punctuation;
• numbers;
• abbreviations;
• names;
• apostrophes and contractions;
• multilingual or code-switched text;
• repeated lyric fragments;
• non-lexical text such as “ah”, “oh” or breath markers.

It is useful to save both the original and normalized text. Otherwise a later failure may look like a G2P error even though the word was changed or removed during normalization.

Pronunciation

The G2P component should output phonemes in a documented canonical inventory. It may use:

• a manually curated dictionary;
• a dictionary plus a G2P fallback;
• a neural G2P model;
• multiple pronunciation candidates;
• language-specific rules;
• a context model for homographs.

Phoneme-set mapping

The phones produced by the G2P system may not match the phones expected by the aligner or the DiffSinger model.

Examples of distinctions that often require explicit mapping include:

silence:
sil / pau / SP

breath:
br / breath / AP

closures or geminates:
cl / q / closure / language-specific phones

The safest pattern is:

external G2P phones
→ explicit mapping table
→ one canonical internal phone set
→ explicit export mapping

Unknown phones should normally produce a visible validation error or review flag rather than silently disappearing.

Lyrics–performance matching

A correct dictionary pronunciation is not enough if the singer performs something different from the supplied lyrics.

Possible mismatches include:

• repeating a word or syllable;
• skipping a word;
• adding an ad-lib;
• humming;
• inserting a breath;
• changing a vowel;
• reducing or dropping a consonant;
• sustaining one vowel across many notes;
• using a pronunciation variant not present in the dictionary.

This layer is important because a single local mismatch can make a strictly monotonic forced alignment drift through the rest of the line.

SOFA has a matching mode that can search for the most probable contiguous subsection of a supplied phoneme sequence rather than always forcing every phone. It also supports confidence output. Those features may be useful for clipped or partially matching segments, although they do not automatically solve every kind of lyric/performance mismatch.

Acoustic alignment

Only after the phone sequence and transcript match are reasonably controlled does it become meaningful to compare acoustic aligners.

Useful controls include:

• Montreal Forced Aligner, as a mature speech-oriented baseline;
• SOFA, as a singing-oriented baseline;
• HubertFA, as another singing-oriented approach;
• a custom WFL/alignment model.

The objective is not necessarily to find one globally best tool. It is to determine which component fails under the target language, phoneme inventory and singing conditions.

A rough map of related projects:

A few points about these comparisons:

MFA

MFA’s documentation provides a useful diagnostic workflow even when MFA is not the final aligner:

validate corpus
→ find OOV words
→ generate/review pronunciations
→ align
→ inspect TextGrids
→ adapt or retrain if needed

The MFA First Steps guide explicitly separates validation, G2P, dictionary generation and acoustic alignment. Its data-validation documentation checks OOVs, unreadable files and transcription/audio mismatches. Its troubleshooting guide also emphasizes verifying that the transcription actually matches the audio.

That makes MFA useful as a control even if a singing-specific model ultimately works better.

SOFA

SOFA’s pipeline is close to the problem described here:

transcription
→ G2P module
→ phoneme sequence
→ singing alignment
→ TextGrid / HTK / transcription output

It supports a custom G2P module, confidence output and a matching mode. This makes it useful both as a baseline and as an example of where the interfaces can be separated.

STARS

STARS demonstrates that a more integrated singing-annotation system is possible. Its alignment interface takes word and phoneme sequences and can predict durations if they are absent; it also predicts notes and singing-style information.

However, it does not remove the pronunciation contract. Its metadata still includes the word list, phoneme list and mapping between them, and the published checkpoints/configurations use particular language-specific phone sets. I would therefore treat it as a later research route rather than the first debugging baseline.

GTSinger

GTSinger provides audio, TextGrid, metadata and realistic music-score information, including manual phoneme-to-audio alignments and paired speech for several languages. If its target language overlaps with this project, it could support a small controlled test such as:

paired speech
vs
natural singing
vs
singing with vibrato/breathy/falsetto/glissando techniques

The repository currently notes that seven of nine languages have refined annotations and that Japanese and Korean are marked for further refinement, so language-specific annotation status should be checked before treating every subset as equally authoritative.

A word-to-phoneme model can be evaluated independently of audio.

The training data for basic G2P is normally:

written form → pronunciation

rather than a waveform alone.

Useful G2P evaluation categories include:

Metrics might include:

• phoneme error rate;
• complete-word pronunciation accuracy;
• OOV coverage;
• accuracy by word category;
• percentage of outputs containing an unknown phone;
• accuracy before and after mapping to the final DiffSinger phoneset.

For OOV handling, a robust engineering baseline is often:

project dictionary
→ language dictionary
→ G2P fallback
→ explicit unknown/review state

rather than making every word depend on one model.

The MFA G2P documentation supports finding OOV words and generating candidate pronunciations, and recommends reviewing generated pronunciations. It can also generate multiple candidates for less transparent orthographies.

Multilingual models such as CharsiuG2P can provide a useful starting point, especially for low-resource languages, but their output phones still need to be checked against the project’s inventory. Similarly, Phonemizer supports several backends—eSpeak, eSpeak-MBROLA, Festival and Segments—and those backends do not necessarily produce interchangeable phones or boundary information.

For reproducibility, I would save:

original text
normalized text
tokenized words
G2P backend/model version
raw G2P phones
phones after mapping
unknown/replaced symbols

This is valuable because silent normalization or tokenization changes can otherwise be mistaken for acoustic-model failures.

Singing differs from ordinary speech alignment in several ways:

• vowels may be sustained for a long time;
• one syllable may span several notes;
• consonants may be extremely short or weakened;
• vibrato and pitch range can change acoustic representations;
• breaths and non-lexical vocalizations are common;
• accompaniment and reverberation may be present;
• the performed lyrics may differ from the written lyrics.

Because of this, the alignment input should contain what was actually performed as closely as possible.

For example, these should not all be treated as the same ordinary silence phone:

leading silence
internal pause
breath
aspiration
humming
unknown vocalization
unmatched audio

Whether they become separate phones, separate tiers or review flags depends on the final DiffSinger/vLabeler convention, but the distinction should be explicit.

If score or note information is available, it can provide useful coarse duration constraints. Singing-alignment research has used hierarchical methods that first estimate syllable or coarse onsets and then refine phoneme boundaries. Therefore a possible score-informed route is:

lyrics + score/note timing
→ approximate word/syllable regions
→ phoneme alignment inside each region

This can be compared with a lyrics-only route rather than assumed to be better in every case.

Accompaniment handling should also be a comparison rather than an unconditional preprocessing rule:

original mixture
separated vocal
clean vocal stem, if available

Source separation may improve alignment by reducing accompaniment, but separation artifacts can also damage consonants and timing cues. A small three-condition comparison is more informative than deciding this in advance.

A useful mismatch test is to deliberately supply one incorrect word or missing syllable and observe whether the system:

• detects low confidence;
• rejects the file;
• confines the damage to a local region;
• skips or repeats phones;
• drifts for the remainder of the line.

For a labeling tool, graceful failure is often more valuable than producing a complete but incorrect label.

“TTS data” can refer to several materially different supervision regimes:

This distinction is more important than the label “TTS”.

For the word-to-phoneme model

The main target is the pronunciation sequence. Audio is not normally required unless the system is also learning pronunciation variants from recordings or using audio to select among candidates.

For the alignment model

Audio is required, but the amount of supervision varies.

A fully labeled example contains:

audio
phoneme sequence
start/end timing for each phoneme

A weakly labeled example may contain:

audio
phoneme sequence
no timing

The latter can still be useful with CTC, monotonic alignment, pseudo-labeling or other weak-supervision approaches, but it does not directly provide a verified boundary target.

Synthetic durations

If the TTS engine exposes internal phoneme durations, those can create inexpensive synthetic timing labels. This may be useful for:

• checking tensor shapes and file-writing code;
• pretraining;
• balancing rare phonemes;
• creating controlled pronunciation examples;
• testing expected phone counts.

However, the generated durations describe the TTS model’s timing behavior. They are not automatically equivalent to manually labeled natural singing boundaries.

A reasonable split would be:

synthetic/TTS set:
development, pretraining, stress tests

small reviewed natural-singing set:
model selection, threshold calibration, final evaluation

It is also important to keep the reviewed evaluation recordings out of any pseudo-label generation or model adaptation that would leak their answers into the system.

One combined score would hide too much.

I would use three evaluation layers.

A. Pronunciation evaluation

Hold audio and timing out of the test.

Measure:

• phoneme error rate;
• word exact match;
• OOV accuracy;
• phone-inventory violations;
• error rates for names, homographs and code-switched words.

B. Alignment evaluation

Use a manually verified phoneme sequence as input.

Measure:

• median boundary error;
• 90th or 95th percentile boundary error;
• percentage of boundaries within 10, 20 and 50 ms;
• catastrophic failures;
• cumulative drift;
• phone-count mismatch;
• zero/negative duration intervals.

The exact manual boundary is not always physically unique, especially around glides, transitions, fricatives and stop closures. Multiple tolerance windows and practical editing measures are therefore usually more informative than treating 0 ms error as the only success condition.

C. End-to-end usefulness

Use the model’s own G2P output and final exporter.

Measure:

• percentage of files imported successfully;
• percentage of phonemes requiring correction;
• percentage of boundaries requiring correction;
• audio minutes per human editing minute;
• number of rejected/flagged files;
• round-trip changes after saving in vLabeler;
• downstream DiffSinger training impact, if that is eventually tested.

The most informative control matrix would be:

This makes error attribution much easier.

Confidence and abstention

I would make confidence or review status a first-class output:

accepted
review recommended
rejected / mismatch suspected

Possible warning signals include:

• low alignment confidence;
• unknown phoneme;
• unexpected phone count;
• extremely short or long phone;
• excessive blank/silence region;
• poor coverage of the supplied transcription;
• large cumulative timing drift;
• duration sum inconsistent with the audio;
• import/export loss.

A threshold should be calibrated on a small reviewed development set rather than copied from another project. The useful operating point depends on whether the goal is high coverage or low manual correction cost.

Dataset splits

For an acoustic model, evaluation should preferably use singers not present in training. Also avoid evaluating pseudo-labels only with the same model that generated them, because that may reward the generator’s own annotation style.

There are several valid routes, depending on the actual goal.

Route A: Engineering-first converter

existing dictionary/G2P
→ explicit phone mapping
→ existing singing aligner
→ TextGrid/timed-lab converter
→ vLabeler review

Best when the immediate goal is a usable labeling assistant.

Required data: a dictionary or G2P model and a small reviewed singing set.

Success criterion: reduced editing time and few catastrophic failures.

Useful baselines: SOFA, HubertFA and MFA.

Route B: Custom G2P research

text
→ WFL/G2P model
→ canonical phones
→ fixed existing aligner

Best when the main novelty is pronunciation, language coverage or lyric-specific text handling.

Required data: spelling–pronunciation pairs and language-specific test categories.

Success criterion: lower pronunciation error, especially for OOVs and lyric-specific forms.

Important control: keep the aligner fixed so alignment differences do not obscure G2P quality.

Route C: Custom singing aligner research

verified phones + audio
→ WFL/alignment model
→ timed phones

Best when pronunciation is already known and the main goal is improved boundary prediction on singing.

Required data: weakly or fully aligned singing recordings.

Success criterion: lower boundary/editing error on held-out singers.

Important control: compare against both a speech-oriented baseline and singing-oriented baselines.

Route D: Integrated annotation system

lyrics/phones + audio (+ optional score)
→ phones, durations, notes and other annotations

This is closer to projects such as STARS.

It is a valid longer-term research direction, but debugging is harder because pronunciation, alignment, transcription and formatting errors interact. I would normally establish Routes A–C as controls first.

Even if the final model is integrated, it is still useful to expose intermediate outputs:

normalized text
phones
word-to-phone mapping
alignment confidence
phone durations
rejected intervals

That makes the system auditable and allows future format converters or model replacements without retraining everything.

A useful initial set does not have to be large. It should be intentionally diverse.

For example:

For each sample, store:

audio
original text
normalized text
verified phones
predicted phones
predicted timing
confidence
expected export
actual vLabeler import result
manual corrections
notes

A result table could look like:

This small table may reveal more than immediately training on a large unreviewed TTS collection.

Before choosing the model, it would help to freeze one exact input/output example.

For example, suppose the existing word-level file is:

0        12000000    hello
12000000 25000000    world

and the desired output is:

0        1800000     HH
1800000  5800000     AH
5800000  8200000     L
8200000  12000000    OW
12000000 13700000    W
13700000 16800000    ER
16800000 19800000    L
19800000 25000000    D

The contract should specify:

For a strict first implementation, useful invariants are:

phones are in the expected order
all durations are positive
boundaries are monotonic
first phone starts at the word start
last phone ends at the word end
the sum of phone durations equals the word duration

Those conditions make the exporter easy to validate.

Later, if word boundaries are treated as soft anchors, the first/last equality conditions can be relaxed within a documented tolerance.

I would also keep a richer internal representation than the final .lab permits:

{
  "word": "singing",
  "word_start": 1.2,
  "word_end": 1.8,
  "phones": ["s", "ih", "ng", "ih", "ng"],
  "pronunciation_source": "dictionary",
  "phone_intervals": [],
  "confidence": null,
  "status": "pending"
}

The final .lab can remain simple, while a sidecar JSON or TextGrid retains diagnostics and confidence.

P0a: Equal-duration split

Divide the word interval equally among its phonemes.

phone duration = word duration / number of phones

This is not an acoustic alignment method, but it is a very useful smoke test for:

• parsing the existing word .lab;
• G2P output;
• phoneme inventory;
• time-unit conversion;
• phone count;
• monotonic boundaries;
• .lab export;
• vLabeler import.

If this baseline does not round-trip correctly, training a better acoustic model will not fix the format pipeline.

P0b: Duration-prior split

A stronger audio-free baseline is to assign relative durations using:

• durations exposed by a TTS engine;
• average duration per phoneme;
• consonant/vowel duration classes;
• onset–nucleus–coda statistics;
• duration distributions measured from labeled singing data.

For example:

phones:       k   ae   t
relative:   0.2  0.6  0.2

The ratios are normalized and scaled to the known word duration.

This remains a prediction of expected timing, not measurement of the recorded performance. It can nevertheless provide:

• a better initializer than equal splitting;
• synthetic training labels;
• a fallback when acoustic alignment fails;
• a duration prior for a later aligner.

P1: Exact per-word acoustic alignment

Treat every existing word interval as a short utterance:

audio[word_start:word_end]
+ expected phone sequence
→ phone intervals

Advantages:

• straightforward implementation;
• failures remain local to one word;
• easy parallelization;
• easy use of existing aligners;
• output naturally fits the word envelope.

Possible problems:

• the supplied boundary may cut a consonant or transition;
• very short words may not contain enough acoustic evidence;
• coarticulation with neighboring words is removed;
• the segment may contain leading/trailing material not represented by the phones.

P2: Padded per-word alignment

Add context around the interval:

audio[word_start - padding : word_end + padding]

Possible padding experiments:

50 ms
100 ms
200 ms
10–20% of the word duration
part of the previous and next word

After alignment:

1. identify the path belonging to the target word;
2. inspect confidence near both edges;
3. remove contextual phones or silence;
4. reconcile the retained phones with the target word interval;
5. flag cases that require excessive boundary movement.

Simply aligning a padded segment and clamping every result to the original word boundary can cause several boundaries to pile up at the edge, so the reconciliation step should be explicit.

P3: Phrase-level alignment

Align the complete phrase using the complete phoneme sequence:

phrase audio
+ phones for all words
+ word-to-phone mapping
→ global phone alignment

Advantages:

• preserves coarticulation;
• avoids repeating the audio encoder for every word;
• gives the decoder more acoustic context;
• allows cross-word transitions.

Risks:

• one incorrect pronunciation can shift later alignments;
• repeated or omitted lyrics can disturb a monotonic path;
• existing word boundaries are no longer automatically preserved.

P4: Phrase-level alignment plus word-anchor reconciliation

First perform global phone alignment, then compare the resulting word groups with the supplied word intervals.

Possible policies include:

hard:
phones are forced back into the supplied word envelope

bounded:
a word boundary may move by at most a fixed tolerance

soft:
moving away from the supplied boundary receives a penalty

review:
large disagreement is preserved and flagged instead of hidden

This is likely the most flexible engineering route, but it needs an explicit policy for disagreement between acoustic evidence and the original word labels.

P5: Custom WFL model

Once P0–P4 provide controls, the WFL model can be evaluated by exactly what it improves:

• pronunciation;
• acoustic phone compatibility;
• boundary localization;
• singing robustness;
• word-anchor use;
• failure detection;
• editing time.

Montreal Forced Aligner

Montreal Forced Aligner can read long audio accompanied by a TextGrid whose intervals contain orthographic transcriptions. Its output contains word and phone tiers.

That means a TextGrid in which each current word interval is treated as a short transcribed interval can provide a direct baseline for:

bounded audio interval + word
→ word and phone tiers

The relevant input structure is described in the MFA corpus-format documentation.

I would describe this carefully: MFA can treat each interval as a bounded utterance, but that is not necessarily the same as giving its decoder immutable word-boundary anchors.

The current documentation also notes that TextGrid intervals shorter than 100 ms are not aligned. This may matter for very short words or incorrectly narrow word labels.

Other considerations:

• MFA is primarily a speech-oriented baseline;
• the pronunciation dictionary and acoustic model must use compatible phones;
• singing performance should be measured rather than assumed;
• word intervals should actually contain the expected sung material;
• extremely short or clipped intervals may need padding or grouping.

Even if MFA is not the final tool, it can reveal whether the file structure, dictionary, and phone sequence are reasonable.

SOFA

SOFA is a singing-oriented forced aligner.

Its normal workflow is close to:

audio + transcription
→ G2P
→ phoneme sequence
→ singing alignment
→ TextGrid / HTK / transcription output

Useful features include:

• dictionary or custom G2P modules;
• TextGrid and HTK output;
• saved confidence scores;
• a matching mode that searches for the most probable contiguous portion of the supplied phoneme sequence instead of requiring every phone.

That matching mode may be useful for clipped word intervals or partial mismatches. It should still be tested with the target language and phoneme set.

The README does not appear to describe a direct interface for treating externally supplied word boundaries as immutable decoder constraints. Possible wrappers are therefore:

one SOFA inference per word interval

or

one inference for the phrase,
followed by word-anchor reconciliation

or

a modification that supplies masks/constraints to inference

HubertFA

HubertFA is another singing-oriented phoneme aligner.

Its evaluation tooling is especially relevant here. It compares predicted and manually annotated TextGrids containing the same phoneme sequence and reports measures such as:

• boundary edit ratio;
• weighted boundary edit ratio;
• a vLabeler-oriented boundary error ratio.

This reinforces the idea that G2P and boundary prediction should be evaluated separately:

verified phones + aligner

before testing:

predicted phones + aligner

The repository also warns that objective scores depend on annotation style, so the numbers are most meaningful when comparing models on the same reviewed dataset.

OpenVPI dataset-tools

The current OpenVPI dataset-tools repository separates related operations into applications, including:

• MinLabel with G2P conversion;
• lyric forced alignment;
• HuBERT phoneme forced alignment;
• TextGrid output;
• breath-label processing.

That modular structure is another useful reference: pronunciation, lyric timing, phoneme timing, breath handling, and export do not have to be one inseparable model.

CTC segmentation

If a CTC acoustic model exists whose vocabulary corresponds to the target phones, ctc-segmentation provides a smaller implementation route:

audio
→ frame-level CTC probabilities
+ expected token/phone sequence
→ monotonic alignment
→ confidence

It can accept an already tokenized sequence and provides confidence scores that can help detect missing or mismatched transcription.

However:

• it requires a compatible CTC model;
• repeated or unrelated segments can disrupt alignment;
• CTC outputs are often temporally peaky;
• token timestamps are not automatically precise phoneme onset and offset boundaries.

I would use it as a prototype or comparison route rather than assuming it produces final vLabeler-quality boundaries.

The older TorchAudio Wav2Vec2 forced-alignment tutorial is useful for understanding the trellis idea, but its page currently warns that the demonstrated APIs were deprecated in TorchAudio 2.8 and scheduled for removal in 2.9. I would not build a long-lived implementation around those exact APIs without checking the current replacement.

There are at least four different boundary policies.

A. Fixed segment window

The supplied word interval defines the audio clip.

word_start ≤ every predicted phone boundary ≤ word_end

This is simple and makes errors local.

The limitation is that the supplied word interval may not perfectly correspond to an acoustic boundary.

B. Fixed output envelope

The aligner operates with context, but its final result is rescaled or adjusted so that:

first phone start = word start
last phone end = word end

This ensures a clean output format, but it can make a poor alignment appear structurally valid. Excessive correction should therefore create a warning.

C. Soft word anchor

The acoustic model may move a boundary slightly, but movement away from the supplied word boundary has a cost.

Conceptually:

alignment score
+ duration prior
- word-boundary displacement penalty

This can handle small annotation errors and coarticulation, but it requires a tolerance or penalty that must be calibrated.

D. Global alignment plus reconciliation

Align the whole phrase, group phones back into words, and compare the inferred word envelopes with the supplied word intervals.

For each word, calculate values such as:

supplied start/end
inferred start/end
start displacement
end displacement
phone confidence
reconciliation status

Then apply a policy:

small disagreement:
accept or softly adjust

moderate disagreement:
keep result but mark review_required

large disagreement:
reject or fall back to per-word alignment

A useful development order would be:

fixed segment
→ padded segment
→ global alignment
→ soft/global reconciliation

One additional detail worth defining is what the current word boundary represents:

• a carefully annotated acoustic boundary;
• an approximate lyric region;
• the interval assigned to the word by another model;
• an interval produced by equal or predicted duration allocation.

The answer changes how strongly the boundary should constrain the phonemes.

If the model receives audio

Possible outputs include:

phone durations
boundary probabilities
frame-to-phone scores
phone posterior probabilities
a monotonic alignment path

A possible abstract architecture is:

audio encoder
    ├─ phone compatibility scores
    └─ boundary/onset scores

expected phone sequence
+ word interval
+ optional duration prior

→ monotonic decoder
→ phone intervals

The model does not need to use this exact architecture, but the separation is useful:

• phone compatibility asks which phone explains a frame;
• boundary probability asks whether a transition occurs at that frame.

Correct phone recognition does not always imply precise boundary localization.

Supervision can also vary.

Fully supervised:

audio
word interval
phone sequence
phone boundaries

Weakly supervised:

audio
word interval
phone sequence
no phone boundaries

The weakly supervised case may use CTC, forward-sum alignment, Viterbi decoding, pseudo-labels, or duration priors. It should not be evaluated only against labels produced by the same model.

If the model does not receive audio

Then its natural target is something like:

word + phoneme sequence
→ relative phoneme durations

This can still be useful as:

• a first-pass label generator;
• a duration prior;
• a fallback;
• an initializer for an acoustic aligner;
• a synthetic-data component.

But it cannot directly observe whether a consonant was omitted, a vowel was extended, or a breath occurred in that particular recording.

A hybrid may be especially practical:

duration-only model proposes initial boundaries
→ acoustic aligner adjusts them
→ confidence/review layer catches failures

A forced aligner can assign a timestamp to every target phone even when the recording contains weak or no acoustic evidence for one of them.

These are not exactly the same claim:

the decoder produced an interval for phone X

the recording clearly supports phone X

Therefore, I would keep confidence and status internally even if the final .lab format cannot store them.

For example:

{
  "phone": "t",
  "start": 1.420,
  "end": 1.455,
  "confidence": 0.18,
  "status": "forced_low_confidence"
}

Possible statuses:

acoustically_supported
forced_low_confidence
recovered_from_prior
boundary_conflict
pronunciation_unknown
review_required
rejected

Useful automatic warning conditions include:

• unknown phoneme;
• word not found in the dictionary;
• extremely short phone;
• extremely long phone;
• zero or negative duration;
• several boundaries collapsing at a word edge;
• poor coverage of the target phone sequence;
• large disagreement with the word boundaries;
• large leading or trailing silence;
• phone-count mismatch;
• low alignment confidence;
• output failing the vLabeler round-trip.

A labeling assistant does not necessarily have to label every word automatically.

A practical output policy may be:

high confidence:
write normally

medium confidence:
write initial label + review marker

low confidence:
use duration fallback or leave for manual labeling

This may save more time than forcing a plausible-looking result for every interval.

SOFA can save confidence values, and CTC segmentation also uses confidence to identify and filter mismatched segments. Their numeric thresholds should not be copied directly; a threshold should be calibrated on a small reviewed sample from the target data.

I would evaluate three separate layers.

Pronunciation

Use no boundary prediction in this test.

Metrics:

• phoneme error rate;
• whole-word exact match;
• OOV accuracy;
• unknown-phone rate;
• pronunciation-variant accuracy;
• error rate after mapping to the final DiffSinger phoneset.

Alignment

Use a manually verified phone sequence.

Metrics:

• median boundary error;
• 90th or 95th percentile error;
• percentage within 10, 20, and 50 ms;
• boundary edit distance or ratio;
• words with catastrophic alignment failure;
• boundaries collapsed at an interval edge;
• phone deletion/insertion behavior;
• confidence calibration.

End-to-end usefulness

Use the complete G2P → alignment → export pipeline.

Metrics:

• percentage successfully imported by vLabeler;
• percentage of phone labels requiring correction;
• percentage of boundaries requiring correction;
• editing time per minute of audio;
• rejected/reviewed word percentage;
• round-trip changes after saving;
• downstream DiffSinger effect, if eventually measured.

A useful control matrix is:

Additional comparisons:

For acoustic-model evaluation, it is better to hold out singers rather than only holding out recordings from the same singers.

It is also useful to preserve the original word labels and predictions rather than overwriting them, so a later model or evaluator can reproduce the comparison.

An initial wrapper could look conceptually like this:

for word_interval in word_labels:
    word = normalize(word_interval.label)

    phones, pronunciation_info = get_pronunciation(
        word=word,
        dictionary=dictionary,
        g2p=g2p_model,
    )

    if not phones:
        record_review_case(
            word_interval,
            reason="no_pronunciation",
        )
        continue

    audio_window = extract_audio(
        audio,
        start=word_interval.start - context_padding,
        end=word_interval.end + context_padding,
    )

    prediction = align_phones(
        audio=audio_window,
        phones=phones,
        word_start=word_interval.start,
        word_end=word_interval.end,
        mode="per_word_padded",
    )

    prediction = reconcile_with_word_anchor(
        prediction,
        word_interval,
        policy="hard",  # later compare with bounded/soft
    )

    validate_prediction(
        prediction,
        expected_phones=phones,
        require_positive_durations=True,
        require_monotonic_boundaries=True,
    )

    output.extend(prediction.phone_intervals)
    diagnostics.append(prediction.diagnostics)

write_phone_lab(output)
write_diagnostics(diagnostics)

A simple roadmap:

Prototype 0:
word-level parser + equal-duration exporter

Prototype 0.5:
dictionary/G2P + duration-prior exporter

Prototype 1:
one existing aligner per exact word interval

Prototype 2:
padded word intervals + reconciliation

Prototype 3:
phrase-level emissions/alignment + word mapping

Prototype 4:
confidence, rejection and review queue

Prototype 5:
custom WFL model compared with all earlier controls

Potential optimization:

encode the phrase audio once
→ reuse frame-level features/emissions
→ decode each word inside its approximate frame range

This preserves context and avoids repeatedly running a large audio encoder for every word.