Translation Fine-tuning assistance

Hi everyone,

I am currently working on fine-tuning models for multi-directional translation tasks—specifically focusing on Japanese–English (ja-en), English–Arabic (en-ar), and Japanese–Arabic (ja-ar).

To ensure the best possible translation quality and prevent common pitfalls (like hallucination or mixed-language outputs), I’m looking to optimize my workflow. Could the community share recommendations on:

  1. Dataset Formatting & Templates: What are the most effective prompt structures or chat templates for translation fine-tuning (e.g., standard source-target pairs vs. instruction-based formatting)?
  2. Instruction Design: Are there specific system prompts or instruction guidelines that yield the best results for low-resource pairs or distant language pairs like Japanese-to-Arabic?
  3. Recommended Base Models: Which open-source base models (e.g., Llama, Mistral, Qwen, or specific multilingual models) have you found to be most robust for these specific language directions?

Any advice, recommended templates, or references to successful repositories would be greatly appreciated!

Thanks in advance!
Edit: I will use the model after fine-tuning in two thing translating general Movies and Tv shows --------- translating Animes

For three directions across three scripts, freeze the format before anything else: one identical template for every example, the target language named in the same place every time, and loss computed only on the translation itself. If the direction signal varies, the model learns to guess the target from the source instead — which is where mixed-language outputs start, worst on ja→ar where the signal is thinnest. Check it directly: on a held-out set scored per direction, count any output in the wrong language (a simple language-ID pass) as a failure before you look at BLEU or chrF, and choose the base model by that off-target rate after the same short fine-tune on a small slice for each candidate.

One implementation detail to add to nootxlm’s suggestion: keep the instruction layout consistent across language directions, but use each chat checkpoint’s own tokenizer chat template when comparing models. Their role/control tokens can differ. For a base checkpoint, first check whether a chat template is provided at all.

If you’re using TRL, putting the translation in an assistant message does not, by itself, ensure translation-only loss. The current SFTTrainer docs distinguish conversational language-modeling datasets from prompt-completion datasets. For assistant-only training, check assistant_only_loss=True and whether the template supports assistant masks ({% generation %} / {% endgeneration %}); check the docs matching your installed version.

Before a full run, inspect one actual collated batch: decode the tokens whose labels are not -100, check that the translation survives truncation, and verify source/padding masking and end-of-turn supervision. If manually formatting training examples, use add_generation_prompt=False; if tokenizing that formatted text separately, use add_special_tokens=False to avoid duplicating special tokens (Transformers docs).

Could you share the exact checkpoint, trainer/version, and one anonymized source-target example? That would let us check the formatting concretely.

Disclosure: I maintain TAF. Its template check can inspect a Hub checkpoint’s template presence/family; it does not inspect your training labels or establish translation quality.