Cannot force Llama-2 to produce the desired spelling of a word

I’ve been fine-tuning Llama-2 using LoRA. The task is to classify the title into correct categories/extract relevant named entities.
However, when I ran the inference, Llama-2 always produces very unstable spellings although it was trained on another spelling.

Example 1:

Input: L’Occitane Premium Hand Cream

Training data:

Beauty products | Hand Cream

Llama-2 output:

Beauty products | Hand Créme

Example 2:

Input: J&J Talcum Powder

Training data:

Brand: J&J

Llama-2 output:

Brand: Johnson & Johnson

Example 3:

Input: Bluetooth Keyboard for Microsoft Surface 2022

Training data:

Electronics | Tablet | Accessories

Llama-2 output:

Electronics | Tablets | Accessaries

Do you know how to force Llama-2 to stay faithful to the input texts’ spellings?

1 Like