I am interested in the topic of fine-tuning a reasoning model. I have a specific idea in mind but no code to share for now.
I have a task where the model will likely benefit from inference-time scaling but this task is not math or code. It requires some fine-grained work with sequences (aligning two or more sequences, comparing correspondences, modifying them, etc.).
In this case, I wonder whether it makes more sense to take a model that was already post-trained for a different reasoning task, like Deepseek-R1 model, or would it rather be better to take a pre-trained model and fine-tune for my specific task, and use some inference-time scaling techniques during fine-tuning?
That’s an interesting problem!
Since your task involves sequence manipulation and comparison, leveraging a pre-trained model is definitely a good starting point. While Deepseek-R1 is trained for reasoning, its strengths might be more geared towards formal reasoning (math, code). For sequence-based tasks, a model pre-trained on a large text corpus (like a standard LLM) could be more beneficial. You can then fine-tune it specifically for your sequence alignment and comparison task. Inference-time scaling techniques can be applied during fine-tuning to improve performance. This approach allows you to benefit from the general language understanding of the pre-trained model while specializing it for your specific needs.