GPT-J generating chatbot response

Hello

Iā€™m using EleutherAI/gpt-j-6B for a chatbot. Iā€™m using the following prompt and the following code:

prompt = "person alpha:\nhi! how are you doing?\n\nperson beta:I am fine, thank you. What are you doing?\n\nperson alpha:\nI am at home watching tv.\n\nperson beta:\nThat sounds like a lot of fun. What are you watching?\n\nperson alpha:\n"

model = GPTJForCausalLM.from_pretrained(
                sEleutherAI/gpt-j-6B,
                revision="float16",
                torch_dtype=torch.float16,
                low_cpu_mem_usage=True,
                use_cache=False,
                gradient_checkpointing=True
        )
tokenizer = transformers.AutoTokenizer.from_pretrained(self.hf_name, pad_token='<|endoftext|>',eos_token='<|endoftext|>')

resp_len = 128
resp_temp = 0.72
prompt = self.tokenizer(text, return_tensors='pt')
prompt = {key: value.to('cuda') for key, value in prompt.items()}
ex_min = len(prompt) + resp_len
out = self.model.generate(**prompt,
                           min_length=ex_min,
                           max_length=ex_min + resp_len,
                           do_sample=True,
                           top_k=35,
                           top_p=0.90,
                           temperature=resp_temp,
                           #repetition_penalty=1.5,
                           #length_penalty=1.2,
                           no_repeat_ngram_size=4,
                           clean_up_tokenization_spaces=True,
)

res = self.tokenizer.decode(out[0])

The output of the model is then the following:

person alpha:
hi! how are you doing?

person beta:
I am fine, thank you. What are you doing?

person alpha:
I am at home watching tv.

person beta:
That sounds like a lot of fun. What are you watching?

person alpha:
A movie called ā€˜The Hobbitā€™. It is a good movie.

Person beta:
Ah, I have not seen that movie. I will look for it.

The above conversation is an example of how the system can handle more than two participants. The conversation can continue with any number of participants.

If person alpha has not been added to the system, the conversation will go through the normal steps of finding the user and adding the person to the system. In the above example, the conversation would continue with person beta.

Figure 4.2: Conversation with more than two people

The conversation can be continued in a number of ways. For example, person beta can start the conversation again. Person beta can start a conversation with person alpha, or person alpha can start a new conversation with person beta, or person beta can join a conversation with someone else.

4.2.2. Conversation and Media

The Conversations system allows media to be used in a conversational context. In this section we will look at how media can be used in the Conversations system.

In the Conversations user interface, media is added to a conversation by dragging and dropping the media file onto the conversation.

When a media file is added to the conversation, the file is added as an attachment to the conversation. This means that the media file can be used by the participants in the conversation. For example the person alpha can add the audio file to the conversation and play it for the person beta. Person beta could also add the video file and play it.
The Conversions system does not use any of the media files for anything other than the conversation. In the Conversations application, the media is just used as a medium for conversation.
The media file is not deleted when the conversation is finished. The file remains on the server and can be used later.

You can add more than one media file to a conversation.
For example, if you

The response of person alpha

A movie called ā€˜The Hobbigā€™. It is a good movie

makes totally sense but then the model continues with person beta and adds much more useless text.

What I want is just the response for person alpha and then the model should stop producing text.

How can this be done? The amount of produced text should also be variable because an answer is not always of same length.

1 Like

Does somebody have an answer here? Any help is appreciated.

assuming person alpha is the AI and person beta is you

  1. get the output as a string variable.
  2. slice the text so it will generate until ā€œperson betaā€.
  3. now it will generate text until it reaches the string ā€œperson betaā€ also in the prompt if you describe your AI describe it first then add conversations between it and you examples