Help with preparing train data for fine-tuning llama 3.1 instruct model?

Models
1

Hi everyone,

I just started learning and want to fine-tune the Llama 3.1 70b-instruct model.
I have the retrieved context and user query, along with a prompt telling the LLM to answer the query using the context.

  1. Is this the right one to prepare query-response pair?
    ‘’‘<|begin_of_text|><|start_header_id|>user<|end_header_id|>
    Answer the user query using the provided context.
    Query:
    {query}
    Context:
    {context}
    Context ends here.
    <|eot_id|><|start_header_id|>assistant<|end_header_id|>
    {response}<|eot_id|>
    "’’
  2. In which format should I combine all query-response pairs together?

Thanks a lot for the help!

1 Like