What can silently go wrong between an SFT JSONL file and the loss tensor?

What can silently go wrong between an SFT JSONL file and the loss tensor?

When an LLM fine-tuning job crashes with a CUDA OOM or produces NaNs, the failure is obvious. The more insidious problems happen when a run completes cleanly with exit code 0, the loss curve descends predictably, but the loss tensor is not optimizing what you intended.

Between raw training records on disk and the computed loss tensor on GPU, a training framework executes a multi-stage transformation:

Raw records → tokenizer/template → collator → model forward
            → shifted labels → CrossEntropyLoss

Because PyTorch only requires compatible tensor shapes, data and masking defects can pass without exceptions. Here are three silent failure modes across that boundary—and how to audit them before spending GPU hours.


1. Truncation starvation — dataset boundary

When an instruction sample exceeds max_seq_len, right-side truncation clips the tail. If a verbose prompt consumes the token budget, the assistant completion can be partially or completely removed.

When the completion is fully dropped, every label in the sequence becomes -100. The sample contributes zero supervised loss while still consuming sequence budget in the batch.

Depending on the chat template, empty completions can similarly result in delimiter-only supervision, effectively training the model to emit closing tokens immediately.

2. Identical preference pairs — alignment boundary

In Direct Preference Optimization (DPO), a row with identical chosen and rejected completions (chosen == rejected) has an implicit reward margin of zero: \Delta r = 0.

Under the deterministic DPO objective, this yields no learning signal. Forward and backward passes still execute, but the pair does not express a preference. Stochastic components such as dropout can introduce residual noise, but do not make such pairs useful preference data.

3. Prompt-loss leakage — tensor-mask boundary

In causal SFT, the intended objective is to condition on the prompt and supervise completion tokens. If a custom collator or template integration fails to mask prompt positions with labels = -100, cross-entropy is calculated over both prompt and answer tokens.

The loss will still decrease, but the objective has changed: the model is also trained to reproduce prompt syntax rather than only generate the intended response.


Catching these early with TrainSight

I built TrainSight (trainsight-preflight), an Apache-2.0 diagnostic package for validating these training contracts. It works at two explicit checkpoints.

1. Preflight inspection — CPU gate

TrainSight audits JSONL datasets—and Parquet datasets when pyarrow is installed—against the selected tokenizer and chat template.

pip install trainsight-preflight

# Check an SFT dataset for truncation starvation and empty completions
trainsight check data/train_sft.jsonl \
  --tokenizer Qwen/Qwen2.5-0.5B-Instruct \
  --max-seq-len 2048

# Check a DPO dataset for identical pairs and length bias
trainsight check data/train_dpo.jsonl \
  --task dpo \
  --tokenizer Qwen/Qwen2.5-0.5B-Instruct

2. In-memory batch reconciliation — tensor invariants

For custom collators or training pipelines, TrainSight audits compiled PyTorch batches. It checks that prompt and padding tokens are masked, and that each active label position (i \ge 1) can contribute to the shifted causal-loss objective.

from transformers import AutoTokenizer
from trainsight import SFTReconciler, LossContract

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

contract = LossContract.from_tokenizer(
    tokenizer,
    max_seq_len=512,
    strict=False,
)

reconciler = SFTReconciler(
    tokenizer=tokenizer,
    contract=contract,
)

# `prompt_lengths` must be derived from the same template/collator path.
report = reconciler.audit_batch(
    batch=batch,
    prompt_lengths=prompt_lengths,
    row_ids=row_ids,
    raw_samples=raw_samples,
)

assert report.passed, report.diagnoses

TrainSight complements existing training frameworks—including TRL, Hugging Face transformers, and native PyTorch loops—by making the relevant loss-contract checks explicit before and during execution.


Discussion

I would value input from practitioners here:

  1. What silent data or loss-masking edge cases have you encountered in SFT or DPO runs that standard validation missed?
  2. How does your team audit collators and template boundaries before launching expensive multi-GPU or multi-node training jobs?

One more that sits past the dataset boundary: packing. When several samples are packed into one sequence without resetting position_ids or blocking attention at the joins, the first tokens of each sample get trained conditioned on the previous sample’s ending — the loss still descends, and a per-file check can’t see it because every row is fine on its own. The cheap test is to pack two short known samples and compare per-token loss against the same samples unpacked: if the second sample’s opening tokens get easier in the pack, the boundary is leaking. Separating attention at the joins (or resetting positions and masking each sample’s first label) fixes it — the packed and unpacked losses on those opening tokens should then agree.

Thanks for laying out these three failure modes so clearly — the truncation one is easy to miss because the run still looks healthy. Two checks I add before a long run are: first, log the fraction of examples where the completion is fully masked or truncated, not just the average length, so a small broken slice cannot hide in the mean. Second, decode a handful of tokenized examples back to text and compare them with the original JSONL, especially around the chat template boundaries, since a template change can silently shift which tokens are supervised. It also helps to run one tiny batch and print the label tensor alongside the decoded tokens before scaling up.