I am currently trying to build an AI Sales Manager that communicates with customers — for now, exclusively via email. It replies to emails, holds conversations with potential clients, and attempts to close deals.
I am using a Qwen model. The problem arose when I put together a small dataset consisting of email conversations from a sales manager who successfully closed deals. I also wrote a system prompt. Essentially, the model operates only on the basis of a playbook, the system prompt, and a small RAG setup.
However, the model still doesn’t communicate the way I need it to. It sometimes duplicates information, occasionally deviates from the sales script and provides slightly different information, may send the same documents multiple times, and — most critically — it sometimes sends payment details to the client for payment even though the client hasn’t confirmed yet or shown any real interest.
How can I fine-tune and refine the AI model specifically for our business’s sales scripts, so that it also learns from the best sales cases and works without the above-mentioned errors?
I would greatly appreciate any advice, suggestions, or guidance on how to approach this properly — whether it involves better data preparation, prompt engineering, fine-tuning strategies, guardrails, or evaluation methods.
Hi Sophie, fine-tuning won’t fix the payment-details one. Even a well-tuned model gets it wrong some % of the time, and for payment details that % has to be zero. That decision needs to come out of the model’s hands.
What I’d do:
Make “send payment details” a tool, not text the model writes. The model can ask for it; your code decides.
Keep a small deal state per conversation in your own code, e.g., NEW > QUALIFIED > CONFIRMED > PAYMENT_SENT. The payment tool only works in CONFIRMED, and CONFIRMED is set by a rule you control (customer explicitly said yes, or a person approves), not by the model. Anything else is denied by default.
Same trick for duplicate documents. Track which docs were already sent in that thread and have the send-document tool refuse a repeat.
For the high-stakes steps, add a human approval for the first few weeks. The agent drafts the payment email, and a person clicks approve. That also gives you real data on where it goes wrong.
Then fine-tune, but for tone and script wording, not for rules. For script adherence, an eval set of a few dozen real past threads where you know the right next step, re-run after every prompt or model change, will tell you more than more training data.
Rules in code, style in the model. That split fixes most of what you listed.
Thanks a lot dear Sattyam. I will definitiely try to implement these approaches into my application and then I will get back with feedbacks.
Appreciate it!
Strong advice from Sattyam above — the only thing I’d add is on the eval side. When you build that script-adherence eval set, make it adversarial: have the test “customer” try to skip stages, ask for payment details early, and contradict things said earlier in the conversation. Then measure stage-transition accuracy and premature-payment rate as separate metrics. Script adherence isn’t one number — the payment-early failure is the one that costs money, so weight it accordingly. Fine-tuning on polite in-script conversations won’t catch any of this; red-teaming the flow will.