How to download a model and run it with Ollama locally?

Beginners
1

I have just installed Ollama on my Macbook pro, now how to download a model form hugging face and run it locally at my mac ?

2

To download a model from the Hugging Face model hub and run it locally using Ollama on your MacBook Pro, you can follow these steps:

  1. Install Ollama: First, ensure that you have Ollama installed on your MacBook Pro. You can download and install Ollama from the official website or through package managers like pip or conda.
  2. Download Model from Hugging Face: Visit the Hugging Face model hub (Models - Hugging Face) and browse or search for the model you want to download. Once you’ve found the desired model, click on it to view its details.
  3. Copy Model URL: On the model’s page, you’ll find a button labeled “Use in Ollama.” Click on this button to copy the URL of the model.
  4. Run Ollama: Open Ollama on your MacBook Pro.
  5. Add Model: In Ollama, there should be an option to “Add Model” or “Import Model.” Click on this option and paste the copied URL of the model from Hugging Face. Ollama should automatically download and add the model to your local environment.
  6. Run Inference: Once the model is added to Ollama, you can use it for inference tasks. Depending on the capabilities of the model (e.g., text generation, image classification), you can provide input data to the model and run inference locally on your MacBook Pro.
  7. Adjust Settings (Optional): Ollama may provide options to adjust settings such as batch size, maximum sequence length, or other model-specific parameters. You can configure these settings based on your requirements.
  8. Run Model: Run the model on your input data using Ollama’s interface. You should see the model’s predictions or outputs based on the input provided.
3

Thank you for your response, is there any other recommend tool to run models locally than Ollama? Why should I use Ollama? Am I on the right track?

4

Certainly! Here are some alternative tools for running models locally:

  1. PyTorch
  2. TensorFlow
  3. ONNX Runtime
  4. OpenVINO
  5. TensorFlow Lite

Each tool offers unique features and advantages, so you can choose based on your specific requirements.

1 Like
5

Thank you for the information, I really appreciate it :smiling_face:

6

Can’t find it… Any other options?

7

You can use git to clone the model to your local storage.
As long as you have git installed on your computer just use the “git clone” command followed by the URL of the model on hugginface.
For example:
git clone mistralai/Mistral-7B-Instruct-v0.2 · Hugging Face

8

So the downloaded model will be in quantized form. Right? Can we use this to fine tune ?

I dont see any config.json file. Any idea ?

9

@hopewise You do realize that both of @KeelyPowers 's answers were generated by a LLM… right?

10

No! But I am not shocked sense this is huggingface forum! :sweat_smile: