I’m working on an AI code reviewer using CodeLlama-Instruct (experimented with 34B f16 and 70B int4) and could use some advice. My setup involves passing a git diff (showing changes with - for removed lines and + for added lines) along with context retrieved via a RAG model to the prompt. Despite explicitly explaining the git diff format in the prompt, CodeLlama seems to misunderstand the concept of diffs and provides confused or irrelevant comments on the code changes.
For example, it might comment on a removed line (-) as if it’s still part of the code or fail to connect the added lines (+) to the intended change. I’ve tried clarifying the diff syntax in the prompt (e.g., “- indicates a line removed, + indicates a line added”), but the issue persists.
Has anyone encountered similar challenges when using CodeLlama-Instruct or other LLMs for code review tasks involving git diffs? I’d appreciate insights on how to solve this problem.
I can share my prompt or example diffs if that would help spark ideas. Thanks for any suggestions or experiences you can share—this community’s expertise is invaluable!
With an LLM of around 70B, there are quite a few tasks that it struggles with, so I think it’s better to format the code with your own script to some extent before passing it on, as this tends to yield more accurate results. Simplify the task itself.
Also, if you’re not set on CodeLlama, you could try other coding models as well.