Mathematic Mistakes

🤗Transformers
1

I am trying to generate some reports using llm.
I tried llama8B,llama70B , Qwen7B

All 3 are having problems in counting. When I pass data, which is a comma-separated data. One task is to count data and work in a count number.
But LLM do a wrong count. For example, I have this data
6 9 13,11 3,0,12 8 9 5,9 9
And LLM has to count each space seprated data , its doing it wrong. This is very small set of data for example, My data is bigger. If data is less then 10 count at a time , its mostly is fine. but when it goes bigger ( counting between 9 and 30) )its start failing and making mistake. And some time it will do wrong count consistently.

Anyone else having this issue.Is it wrong expectation that LLM can count the input data?

Any suggestions for this issue. Any particular fine-tuning can make it better?

Regards,
Abhishek

2

Language models are typically not great at mathematics as they have been trained in language, not so much in mathematics. That problem can be solved easily with classical programming without the help of a language model.