Getting repetative response using ConversationalRetrievalChain + HugginFaace

Beginners
1

Hello I’m Trying To move from Chat GPT api to Hugging Face using ConversationalRetrievalChain
When Getting final answer From llm I’m getting repeated same message which is very weird
Here is example of the code
async def create_crc_chain(self, user_id, repo: UserRepo):
“”“Creatates Chain For User and stores it in SimpleChainManager”“”
# Creating session
# async with self.session_pool() as session:
# repo = RequestsRepo(session)
# await repo.basic.get_or_create_session(user_id)

    # ---Old tamplate solution----
    # prompt = PromptTemplate(
    #     input_variables=["question"],
    #     # , "twitter_information"],
    #     template=SYSTEM_PROMPT,

    # )

    prompt = PromptTemplate.from_template(SYSTEM_PROMPT)

    ######  Setting Multiquery retriever as base retriver ######
    QUERY_PROMPT = PromptTemplate(
        input_variables=["question"],
        template="""You are an AI language model assistant.\n
    Your task is to generate 3 different versions of the given user question to retrieve relevant documents from a vector  database.\n 
    By generating multiple perspectives on the user question, your goal is to help the user overcome some of the limitations of distance-based similarity search.\n
    Provide these alternative questions separated by newlines.\n
    Original question: {question}"""
    )

    document_prompt = PromptTemplate(
        input_variables=["page_content", "source"],
        template="Context:\nContent:{page_content}\n Source file name:{source}",
    )

    condense_question_prompt = PromptTemplate.from_template(QUESTION_PROMPT_TEMPLATE)
    model = "meta-llama/Meta-Llama-3-70B-Instruct"
    hb_token = os.getenv('HUGGINGFACEHUB_API_TOKEN')
    llm = HuggingFaceEndpoint(
        repo_id=model,
        huggingfacehub_api_token=hb_token,
        temperature=0.4,
    )
    document_llm_chain = LLMChain(
        llm=llm,
        prompt=prompt,
        callbacks=None,
        verbose=True
    )

    combine_documents_chain = StuffDocumentsChain(
        llm_chain=document_llm_chain,
        document_variable_name="context",
        document_prompt=document_prompt,
        callbacks=None,
        verbose=True
    )

    base_retriever = MultiQueryRetriever.from_llm(
        retriever=self.docsearch.as_retriever(
            search_kwargs={"k": 20}
        ),
        prompt=QUERY_PROMPT,
        llm=llm
    )

    compressor = CohereRerank(top_n=1, cohere_api_key='T8j5GzGyw0SPCm8wv8ewwEYDsCALVU7lw3EnmwdY')
    # compression_retriever = ContextualCompressionRetriever(
    #     base_compressor=compressor, base_retriever=self.docsearch.as_retriever(search_kwargs={"k": 20})
    # )
    compression_retriever = ContextualCompressionRetriever(
        base_compressor=compressor, base_retriever=base_retriever
    )

    question_generator_chain = LLMChain(
        llm=llm,
        prompt=condense_question_prompt,
        verbose=True)

    memory = await self.create_memory_with_history(
        repo, user_id,
        llm=llm,
    )

    qa = ConversationalRetrievalChain(
        combine_docs_chain=combine_documents_chain,
        question_generator=question_generator_chain,
        callbacks=None,
        verbose=True,
        retriever=compression_retriever,
        return_source_documents=True,
        memory=memory
    )

Here is example of input output
input : hi
output : Hi! I’m here to help you navigate the world of dating billionaire men. I can assist you in having efficient conversations with them and potentially leading to successful relationships. What would you like to know or discuss? Do you have a specific question or situation in mind? Feel free to ask, and I’ll do my best to provide you with personalized guidance and advice.assistant

Hi! I’m here to help you navigate the world of dating billionaire men. I can assist you in having efficient conversations with them and potentially leading to successful relationships. What would you like to know or discuss? Do you have a specific question or situation in mind? Feel free to ask, and I’ll do my best to provide you with personalized guidance and advice.assistant

Hi! I’m here to help you navigate the world of dating billionaire men. I can assist you in having efficient conversations with them and potentially leading to successful relationships. What would you like to know or discuss? Do you have a specific question or situation in mind? Feel free to ask, and I’ll do my best to provide you with personalized guidance and advice.assistant
etc…

What can be issue and maybe you have some thought how I can fix it