Load the model fails for sentence-transformers/sentence-t5-xl

Beginners
1

Trying to load the model ‘sentence-transformers/sentence-t5-xl’

model = SentenceTransformer(‘sentence-transformers/sentence-t5-xl’)
tmp_dir = “sentence-t5-xl”
model.save(tmp_dir)
import boto3
s3 = boto3.client(‘s3’)
bucket = “bucket_name”
s3_prefix = “models/sentence-t5-xl”
import os
for root, _, files in os.walk(tmp_dir):
for file in files:
local_path = os.path.join(root, file)
s3_path = os.path.join(s3_prefix, os.path.relpath(local_path, tmp_dir))
s3.upload_file(local_path, bucket, s3_path)

print(f"Model uploaded to s3://{bucket}/{s3_prefix}")

model saved but failed to load

def download_s3_directory(bucket_name, s3_prefix, local_dir):
# Create an S3 client
s3_client = boto3.client(‘s3’)

# List objects in the S3 directory (prefix)
response = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=s3_prefix)

import os
# Ensure the local directory exists
if not os.path.exists(local_dir):
    os.makedirs(local_dir)

# Iterate over the files and download them
for obj in response.get('Contents', []):
    s3_key = obj['Key']
    file_name = os.path.basename(s3_key)  # Get the file name from the S3 key
    local_file_path = os.path.join(local_dir, file_name)

    if not file_name:  # Skip directories in S3
        continue

    print(f"Downloading {s3_key} -> {local_file_path}...")
    s3_client.download_file(bucket_name, s3_key, local_file_path)

def load_model_from_s3(bucket, model_path, local_dir=“/tmp/sentence-t5-xl”):
import os
if not os.path.exists(local_dir):
download_s3_directory(bucket, model_path, local_dir)
os.environ[“TRANSFORMERS_OFFLINE”] = “1”
os.environ[“HF_DATASETS_OFFLINE”] = “1”
os.environ[“HF_HUB_OFFLINE”] = “1”
import torch
torch.set_grad_enabled(False)
return SentenceTransformer(local_dir, local_files_only=True)

model = load_model_from_s3(bucket, sbert_model_path)

shows the error
modules, self.module_kwargs = self._load_sbert_model(
File “/home/hadoop/environment/lib64/python3.9/site-packages/sentence_transformers/SentenceTransformer.py”, line 1736, in _load_sbert_model
module_path = load_dir_path(
File “/home/hadoop/environment/lib64/python3.9/site-packages/sentence_transformers/util.py”, line 1399, in load_dir_path
repo_path = snapshot_download(**download_kwargs)
File “/home/hadoop/environment/lib64/python3.9/site-packages/huggingface_hub/utils/_validators.py”, line 114, in _inner_fn
return fn(*args, **kwargs)
File “/home/hadoop/environment/lib64/python3.9/site-packages/huggingface_hub/_snapshot_download.py”, line 219, in snapshot_download
raise LocalEntryNotFoundError(
huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass ‘local_files_only=False’ as input.

Not sure what is the problem
sentence-transformers==3.3.1

2 Likes
2

I can’t reproduce the bug… Is it a problem specific to S3, or is it a problem with dependencies…?

import os
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/sentence-t5-xl")
tmp_dir = "sentence-t5-xl"
tmp_dir_dummy = "sentence-t5-xl-dummy"
model.save(tmp_dir)
print(model)
# SentenceTransformer(
#  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: T5EncoderModel ...

model2 = SentenceTransformer(tmp_dir, local_files_only=True)
print(model2)
# SentenceTransformer(
#  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: T5EncoderModel ...

model3 = SentenceTransformer(tmp_dir_dummy, local_files_only=True)
# OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like sentence-transformers/sentence-t5-xl-dummy is not the path to a directory containing a file named config.json.

os.makedirs(tmp_dir_dummy, exist_ok=True)
model4 = SentenceTransformer(tmp_dir_dummy, local_files_only=True)
# OSError: Error no file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt.index or flax_model.msgpack found in directory sentence-t5-xl-dummy.

It seems that in some cases, the problem can be fixed by updating the huggingface_hub library.

pip install -U huggingface_hub
1 Like
3

it was working for few days and now started failing. Can you show me a way to load the model from S3 instead of SentenceTransformer(“sentence-transformers/sentence-t5-xl”)
If I save it to s3 and load it from S3 I am getting lot of error and not able to load the load the model properly.
If I use SentenceTransformer(“sentence-transformers/sentence-t5-xl”) for inference then requests.exceptions.ReadTimeout: (ReadTimeoutError(“HTTPSConnectionPool(host=‘cdn-lfs.hf.co’, port=443): Read timed out. (read timeout=10)”), ‘(Request ID: 0af29d71-521b-4a4e-bbfc-eed6c171e671)’)

2 Likes
4

Although we don’t know which is the real cause, the actual process of downloading from the Hugging Face Hub is usually handled by the huggingface_hub library, so if you can solve or avoid the error in that part, you can often get around it.

Well, there are also individual library cache issues from time to time…

1 Like
5

I would appreciate if you can show me the right way to load the model from S3 instead of
model = SentenceTransformer(“sentence-transformers/sentence-t5-xl”), It would be Like loading from pretrainted model.

  1. Save the model to S3
  2. Download the model from S3
  3. Load it for infernce
2 Likes
6

I don’t know how S3 behaves…

There are also the following ways to download the Hugging Face model.

by Hugging Chat

To load a Sentence Transformer model from Amazon S3, you can follow these steps:

Step 1: Save the Model to a Directory

First, save your pretrained Sentence Transformer model to a local directory.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/sentence-t5-xl")
model.save("saved_model")

Step 2: Upload the Model to S3

Use boto3 to upload the saved model files to an S3 bucket.

import boto3
import os
from botocore.exceptions import NoCredentialsError

def upload_to_s3(bucket_name, local_path, s3_prefix):
    s3 = boto3.client('s3')
    for root, dirs, files in os.walk(local_path):
        for file in files:
            local_file = os.path.join(root, file)
            s3_key = os.path.join(s3_prefix, os.path.relpath(local_file, local_path))
            try:
                s3.upload_file(local_file, bucket_name, s3_key)
                print(f"Uploaded {local_file} to s3://{bucket_name}/{s3_key}")
            except NoCredentialsError:
                print("Credentials not available")
                return False
    return True

# Replace with your bucket name and S3 path
upload_to_s3("your-bucket-name", "saved_model", "path/to/model")

Step 3: Download the Model from S3

When you need to load the model, download the files from S3 into a temporary directory.

import tempfile
from contextlib import contextmanager

@contextmanager
def download_model_from_s3(bucket_name, s3_prefix, temp_dir):
    s3 = boto3.client('s3')
    try:
        # List all objects under the prefix
        paginator = s3.get_paginator('list_objects_v2')
        pages = paginator.paginate(Bucket=bucket_name, Prefix=s3_prefix)
        
        for page in pages:
            for obj in page['Contents']:
                key = obj['Key']
                local_path = os.path.join(temp_dir, os.path.relpath(key, s3_prefix))
                os.makedirs(os.path.dirname(local_path), exist_ok=True)
                s3.download_file(bucket_name, key, local_path)
                print(f"Downloaded {key} to {local_path}")
        yield temp_dir
    finally:
        # Clean up the temporary directory
        shutil.rmtree(temp_dir, ignore_errors=True)

Step 4: Load the Model for Inference

Use the temporary directory to load the model.

with tempfile.TemporaryDirectory() as temp_dir:
    with download_model_from_s3("your-bucket-name", "path/to/model", temp_dir) as model_dir:
        model = SentenceTransformer.from_pretrained(model_dir)
        # Use the model for inference
        embeddings = model.encode("Your text here")
        print(embeddings)

Summary

  1. Save the Model: Use model.save() to save the model to a local directory.
  2. Upload to S3: Use boto3 to upload the entire directory to your S3 bucket.
  3. Download from S3: When needed, download the model files into a temporary directory.
  4. Load the Model: Use the temporary directory to load the model for inference.

This method ensures that all necessary model files are correctly handled, and the temporary directory is cleaned up automatically. Make sure your AWS credentials are properly configured to allow access to the S3 bucket.

References

1 Like
7

Also by Hugging Chat.

The error occurs because the model cannot find the necessary files in the local cache and is prohibited from downloading online. To resolve this, you need to ensure all required files are present in the correct directory and adjust how the model is loaded. Here’s how to fix it:

  1. Inspect the Model Structure: Verify that all necessary files (e.g., config.json, model.bin, tokenization.json) are included when saving the model.

  2. Modify Cache Directory: Set the TRANSFORMERS_CACHE environment variable to point to the local directory where the model is saved.

  3. Adjust Loading Code: Ensure that the model loads from the correct local directory without relying on the default cache.

Here’s the fixed code:

import os
from sentence_transformers import SentenceTransformer

def load_model_from_s3(bucket, model_path, local_dir="/tmp/sentence-t5-xl"):
    # Ensure necessary imports and settings
    import boto3
    from sentence_transformers import SentenceTransformer

    # Create S3 client
    s3_client = boto3.client('s3')

    # Download the model from S3 to local_dir
    if not os.path.exists(local_dir):
        os.makedirs(local_dir)
    
    # List objects in the S3 directory
    response = s3_client.list_objects_v2(Bucket=bucket, Prefix=model_path)
    
    for obj in response.get('Contents', []):
        s3_key = obj['Key']
        file_name = os.path.basename(s3_key)
        local_file_path = os.path.join(local_dir, file_name)
        
        if not file_name:
            continue
        
        print(f"Downloading {s3_key} -> {local_file_path}...")
        s3_client.download_file(bucket, s3_key, local_file_path)
    
    # Set environment variables
    os.environ["TRANSFORMERS_OFFLINE"] = "1"
    os.environ["HF_DATASETS_OFFLINE"] = "1"
    os.environ["HF_HUB_OFFLINE"] = "1"
    os.environ["TRANSFORMERS_CACHE"] = local_dir  # Point cache to the local directory
    
    # Load the model from the local directory
    return SentenceTransformer(local_dir, local_files_only=True)

# Usage example
bucket_name = "your-bucket-name"
s3_model_path = "models/sentence-t5-xl"
model_load_dir = "/tmp/sentence-t5-xl"

model = load_model_from_s3(bucket_name, s3_model_path, model_load_dir)

Explanation:

  • Environment Variable Adjustment: By setting TRANSFORMERS_CACHE to point to the local directory, we ensure the model looks for files in the correct location.
  • Downloading All Files: The code ensures all necessary files are downloaded from S3 to the local directory.
  • Loading Correctly: The model is loaded with local_files_only=True to prevent online downloads and using the specified directory.

This should resolve the LocalEntryNotFoundError by ensuring the model finds all required files locally.

1 Like
8

SentenceTransformer.from_pretraine - It throws error saying SentenceTransformer has no method named from_pretrained. I also used Hugging Chat, I am not successful in loading the model.

2 Likes
9

I think it’s probably a version problem with the hugingface_hub library. It’s not the version of the sentence_transformers, but that version that changes the save and load (serialization, deserialization, downloading, uploading, cache management, token management…)…

In that case, the fastest thing to do is to downgrade.

The next thing to consider is changing the default permission settings on the cloud service side. In this case, you can sometimes deal with it by changing the temporary folder by changing things like HF_HOME.

Anyway, it’s certain that something has changed between when it was working and now when it’s not working. The only thing that could have changed within the scope of what we can do easily here is the versions of the libraries.

If it’s a change on the cloud service side, it’s going to be quite difficult, but in that case, I think someone else might have noticed…

1 Like
10

So if I read the first error message here, it’s going to download and fail…
I think it didn’t go to download if it was the same code before.

Incidentally, if you follow the source, it seems that the original download method uses hf_hub_download.

1 Like
11

There are two ways to download →

  1. model = SentenceTransformer(‘sentence-transformers/sentence-t5-xl’)
    tmp_dir = “sentence-t5-xl”
    model.save(tmp_dir)
  2. huggingface-cli download sentence-transformers/sentence-t5-xl --local-dir sentence-t5-xl
    I tried uploading these file to S3 and while inferencing refering that to avoid connection issue with the huggingface

Both save the model differently.
download_s3_directory(bucket, model_path, local_dir)
model = SentenceTransformer(local_dir)
shows the error for cache files, even if I specify cache to refer local dir it doesn’t work
return SentenceTransformer(local_dir, cache_folder=local_dir, local_files_only=True)

2 Likes
12

shows the error for cache files, even if I specify cache to refer local dir it doesn’t work

Actually, a similar error was reported on HF Discord yesterday, and in that case, only the Transformer model downloaded from git did not work. He had apparently worked around the problem by doing from_pretrained and save_pretrained beforehand…

Could it be that the cause of the error is the same…?

My suspicion is that it doesn’t work properly if there are any additional files, because it doesn’t happen with a repo that has a simple structure with only a single model…

1 Like
13

Can you give the reference ?

2 Likes
14

Can you give the reference ?

2 Likes
16

Error one: deepseek-ai/DeepSeek-R1
No problem one: HuggingFaceTB/SmolLM2-135M-Instruct

1 Like
17

Hmm… Perhaps this commit? You may be able to get around this problem for the time being by setting the cache_folder argument or the SENTENCE_TRANSFORMERS_HOME environment variable.

1 Like
18

Thank you John6666
os.environ[‘TRANSFORMERS_OFFLINE’] = ‘1’
os.environ[‘HF_DATASETS_OFFLINE’] = ‘1’
os.environ[‘HF_HUB_OFFLINE’] = ‘1’
helped to continue.
Instead of downloading it, I did created the zipped version of the model and uploaded to S3 and referred same using
–conf spark.archives={model_path}/sentence-t5-xl.zip#sentence-t5-xl
–conf spark.driverEnv.SENTENCE_TRANSFORMERS_HOME=./sentence-t5-xl
–conf spark.executorEnv.SENTENCE_TRANSFORMERS_HOME=./sentence-t5-xl
then
model = SentenceTransformer(model_name_or_path=‘sentence-t5-xl’, device=‘cpu’, local_files_only=True) resolved the issue.

2 Likes