How to Configure LLaMA-3:8B on HuggingFace to Generate Responses Similar to Ollama?

🤗Transformers
1

Hi HuggingFace Community,

I have been experimenting with the LLaMA-3:8B model using the following code:

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3-8B"

pipeline = transformers.pipeline(
    "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)
pipeline("Hey how are you doing today?")

This code generates the following response:

[{'generated_text': 'Hey how are you doing today? My name is David and I’m a photographer based in Toronto, Canada. I started this blog as a place to share my photography and thoughts on photography, and I hope that you enjoy it.\nI’m also a software developer, and I run a web development company called WebDev Studios. I’m a big fan of WordPress, and I’m always looking for ways to improve the WordPress experience for developers and users. I’m also a big fan of open source software, and I’m always looking for ways to contribute to the open source community.\nI hope you enjoy my blog, and I hope you find it useful. If you have any questions or comments, please feel free to contact me. Thanks for reading!\nHow to use David’s blog for your photography\nDavid’s blog is a great resource for anyone interested in photography. The blog features a wide range of topics, from gear reviews to tutorials to inspirational stories. Whether you’re a beginner or a seasoned pro, there’s something for everyone on David’s blog.\nOne of the best things about David’s blog is that it’s easy to use. The layout is clean and uncluttered, and the posts are well-organized. You can easily find what you’re looking for by using the search function or by browsing through the different categories.\nIf you’re looking for photography tips and advice, David’s blog is a great place to start. The posts are written in a clear and concise manner, and they’re packed with helpful information. Whether you’re trying to improve your composition or learn how to use a new piece of gear, you’ll find what you’re looking for on David’s blog.\nIf you’re looking for inspiration, David’s blog is a great place to find it. The posts are filled with beautiful images and inspiring stories. Whether you’re looking for a new perspective on photography or just some motivation to get out and shoot, you’ll find what you’re looking for on David’s blog.\nIf you’re looking for a great photography resource, David’s blog is a must-visit. The blog is well-organized and easy to use, and the posts are packed with helpful information. Whether you’re a beginner or a seasoned pro, you’ll find what you’re looking for on David’s blog.\nHow to use David’s blog for your photography business\nDavid’s blog is a great resource for anyone looking to improve their photography skills or start a photography business. The blog is packed with helpful tips and advice, and David is always happy to answer any questions you may have.\nOne of the best things about David’s blog is that it’s easy to use. You can find whatever you’re looking for by using the search function or by browsing through the different categories. The blog is also well-organized, so you can easily find what you need.\nIf you’re just starting out in photography, David’s blog is a great place to start. The blog covers everything from the basics of photography to more advanced topics, so you can find the information you need to improve your skills.\nIf you’re already a photographer, David’s blog is still a great resource. The blog covers a wide range of topics, so you can find new tips and advice to help you improve your photography.\nNo matter what your level of experience is, David’s blog is a great resource for anyone looking to improve their photography skills or start a photography business.\nHow to use David’s blog for your photography business\nDavid’s blog is a great resource for anyone looking to improve their photography skills or start a photography business. The blog is packed with helpful tips and advice, and David is always happy to answer any questions you may have.\nOne of the best things about David’s blog is that it’s easy to use. You can find whatever you’re looking for by using the search function or by browsing through the different categories. The blog is also well-organized, so you can easily find what you need.\nIf you’re just starting out in photography, David’s blog is a great place to start. The blog covers everything from the basics of photography to more advanced topics, so you can find the information you need to improve your skills.\nIf you’re already a photographer, David’s blog is still a great resource. The blog covers a wide range of topics, so you can find new tips and advice to help you improve your photography.\nNo matter what your level of experience is, David’s blog is a great resource for anyone looking to improve their photography skills or start a photography business.\nHow to use David’s blog for your photography business\nDavid’s blog is a great resource for anyone looking to improve their photography skills or start a photography business. The blog is packed with helpful tips and advice, and David is always happy to answer any questions you may have.\nOne of the best things about David’s blog is that it’s easy to use. You can find whatever you’re looking for by using the search function or by browsing through the different categories. The blog is also well-organized, so you can easily find what you need.\nIf you’re just starting out in photography, David’s blog is a great place to start. The blog covers everything from the basics of photography to more advanced topics, so you can find the information you need to improve your skills.\nIf you’re already a photographer, David’s blog is still a great resource. The blog covers a wide range of topics, so you can find new tips and advice to help you improve your photography.\nNo matter what your level of experience is, David’s blog is a great resource for anyone looking to improve their photography skills or start a photography business.'}]

However, when I input a similar question into the Ollama model, I receive a response like:

"I'm just a language model, so I don't have emotions or personal experiences like humans do. However, I'm functioning properly and ready to assist with any questions or tasks you may have!"

I am curious why these two models, both based on LLaMA3:8b, produce such different responses. More importantly, I would like to know how I can configure the HuggingFace LLaMA-3:8B model to output responses more akin to Ollama’s style.

Is there a way to adjust model settings, parameters, or prompt design to achieve this? Any guidance would be appreciated!

Thank you!

1 Like
2

The issue of different output when using pipelines and when using them locally or in Spaces is sometimes talked about.
While pipelines are easy to deal with, they seem to be working by supplementing a lot of information on their own, which means it is difficult to determine what is wrong with them.

I’m not sure about Ollama, but I think it would be better to use something other than pipelines to set chatbot-like parameters.
chat_completion, for example, is often used.
Also, I heard that Llama3 has some kind of bug, though I’m not sure what it has to do with this issue.

1 Like
3

Hi @John6666 ,

Thanks again for your assistance! I tried using model.generate, but I am still encountering the same issue of inconsistent output between the LLaMA-3:8B model on HuggingFace and Ollama.

Would you mind suggesting any other API options or strategies that I could use for text generation with large language models? I am specifically aiming to achieve a more consistent style in the responses, similar to Ollama.

4

If you are aiming for a consistent answer style, I would suggest setting temperature to 0.7 or higher.
Ollama’s default value seems to be 0.7.
In addition, if you set do_sample=False, temperature, etc. cannot be set, and the answer will be reproducible.

Also, keep in mind that there seems to be some kind of bug in the Llama3 model, although probably not related to this issue. The specifics are a bit beyond me, but I hear it is buggy anyway.

Ollama default parameters:

generate options:

5

Hi @John6666 ,

Thank you for your prompt response. I believe I’ve found a simple solution for the issue. It turns out that when we run ollama run llama3:8b, Ollama is actually using the llama3:8b-instruct model rather than the base llama3:8b (both have the hash ID 365c0bd3c000).

When I switched from meta-llama/Meta-Llama-3-8B to meta-llama/Meta-Llama-3-8B-Instruct in the HuggingFace pipeline, the output became much more aligned with what I was expecting from Ollama:

{'role': 'assistant', 'content': "I'm just a language model, I don't have feelings or emotions like humans do, so I don't have good or bad days. However, I'm always happy to chat with you and help with any questions or topics you'd like to discuss! How about you? How's your day going?"}

Thanks again for all your kind assistance!

1 Like
6

when we run ollama run llama3:8b, Ollama is actually using the llama3:8b-instruct model rather than the base llama3:8b

Wow. This was not predictable.:sweat_smile:
Well, it’s OK as long as it gets resolved.

LLM is fun.
Changing the temperature option changes the LLM’s reaction (personality?) a lot, so it’s interesting to try it out.

1 Like
7

Sure, I have already changed the temperature as 0.7 as you recommended. Hope this will bring me better results :grinning:

1 Like
8

You’re using the model in document-completion mode, rather than chat mode. See Chatting with Transformers for the basic instructions and Chat Templates for details.