Simple Model to rewrite/paraphrase

Beginners
1

Hey,

I am searching for a model, that can be used for re-writing a text using in a sophisticated style and is as small as possible (should focus only on this task).

I was trying to use the the T5, BART and PEGASUS model but the first two did not change the text while the later gave a completely different text.

The paraphrase models seem to map sentences and paragraphs to dense vectors instead of creating new sentences.

from transformers import PegasusForConditionalGeneration, PegasusTokenizer
source_path  = "/media/admin_ud/Volume/huggingface_cache/huggingface/hub"
model     = PegasusForConditionalGeneration.from_pretrained("google/pegasus-xsum",cache_dir = source_path)
tokenizer = PegasusTokenizer.from_pretrained("google/pegasus-xsum",cache_dir = source_path)

# Input sentence
sentence  = "I have backpain. And I have a headache. And I have pain in my leg."

# Tokenizing the input
input_text = f"paraphrase: {sentence}"
inputs     = tokenizer(input_text, return_tensors="pt", max_length=512, truncation=True)

# Generating reformulated sentence
outputs    = model.generate(inputs["input_ids"], max_length=128, num_beams=5, early_stopping=True)

# Decoding the output
reformulated_sentence = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(reformulated_sentence) # "I have pain in my leg."
``

Which model/model class is suitable for that task?
1 Like
2

PEGASUS is an LM for summarization, so I think its behavior is correct. For tasks like rewriting sentences, I think it would be easier to use a small LLM.

Based on your requirements and the sources provided, here is an analysis of the situation and suggestions for a suitable model:

Why T5, BART, and PEGASUS Might Not Be Suitable

  1. T5: While T5-Small is a compact model (~60 million parameters) designed for various NLP tasks, including text rewriting, it relies heavily on proper fine-tuning and prompting [2]. If you are using it for text rewriting without fine-tuning or with the wrong prompts, it may not produce the desired sophisticated rewrites.

  2. BART: BART is also a text-to-text model that can handle rewriting tasks but might struggle with generating sophisticated paraphrases if it has not been explicitly trained or fine-tuned for this purpose [3].

  3. PEGASUS: PEGASUS is primarily designed for summarization, which involves extracting key information rather than preserving the full context or style of the original text. This explains why it might produce rewrites that are too different from the original.

  4. Paraphrase Models: Many paraphrase models focus on generating paraphrases by mapping sentences to dense vectors, which is not ideal for creating sophisticated rewrites [3].

Recommended Models for Sophisticated Text Rewriting

If the above models are not suitable, here are some alternative models you can explore on Hugging Face:

  1. FLAN-T5: A variant of T5 that has been fine-tuned on a wide range of tasks, including rewriting and paraphrasing. It is instruction-agnostic and can generate more sophisticated outputs when given clear prompts [3].

  2. Instruction-Tuned Models: Models like Mixtral, Cohere Command R+, or Meta Llama3 are designed to follow instructions and generate high-quality text. These models can be fine-tuned for sophisticated text rewriting [3].

  3. Brio or Other Paraphrase Models: Models like Brio or [MBart](https://huggingface.co/facebook/mbart-large-5镘are designed for paraphrasing and can be adapted for text rewriting. However, they may not generate as sophisticated outputs as the instruction-tuned models mentioned above.

Conclusion

For your task, I recommend using FLAN-T5 or an instruction-tuned model like Mixtral. These models are better at following specific instructions and generating sophisticated rewrites. If you are looking for a smaller model, T5-Small can still work if you provide clear prompts or fine-tune it on a dataset with sophisticated paraphrasing examples [2][3].

2 Likes
3

This appears to be the answer from Chat-GPT, since it is the links are wrong and the answer is quite vague

1 Like
4

The second half is a general discussion using Hugging Chat. It’s not as smart as ChatGPT. The first half is manual. I left it to the chatbot to explain why that model was unsuitable for that task, as it was too much trouble to explain.

1 Like
5

Thank you for your part! The problem is, that general models tend to add their own information to the text and this needs to be prohibited in the use case.

That’s why a specialized model would be great, that is trained to not change the meaning of the text or only make minor changes.

1 Like
6

The Instruct models are tuned for chatbot-like use, so I think using the Base models would be a little better, but that tendency is certainly strong in LLM in general. I think something that creates something…
something that’s about halfway between LM and LLM would be good.

7

Thanks so much for this informative response

1 Like