I have deployed an endpoit with mistralai/Mistral-7B-v0.1, and was able to send requst to three days ago. But today when I tryied to resume the endpoint it kept failing, returning download error.
What can be the problem? Thank you!
the following is what I get:
2023/12/14 13:51:12 ~ {“timestamp”:“2023-12-14T05:51:12.753891Z”,“level”:“ERROR”,“fields”:{“message”:“Download encountered an error: Traceback (most recent call last):\n\n File "/opt/conda/bin/text-generation-server", line 8, in \n sys.exit(app())\n\n File "/opt/conda/lib/python3.9/site-packages/text_generation_server/cli.py", line 156, in download_weights\n local_pt_files = utils.weight_files(model_id, revision, ".bin")\n\n File "/opt/conda/lib/python3.9/site-packages/text_generation_server/utils/hub.py", line 90, in weight_files\n raise FileNotFoundError(\n\nFileNotFoundError: No local weights found in /repository with extension .bin\n\n”},“target”:“text_generation_launcher”,“span”:{“name”:“download”},“spans”:[{“name”:“download”}]}
2023/12/14 13:51:41 ~ {“timestamp”:“2023-12-14T05:51:41.931879Z”,“level”:“INFO”,“fields”:{“message”:“Starting download process.”},“target”:“text_generation_launcher”,“span”:{“name”:“download”},“spans”:[{“name”:“download”}]}
2023/12/14 13:51:41 ~ {“timestamp”:“2023-12-14T05:51:41.931740Z”,“level”:“INFO”,“fields”:{“message”:“Args { model_id: "/repository", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 4096, max_total_tokens: 4097, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "sxresearch16b4-mistral7b-768978877c-xv4vv", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: true, otlp_endpoint: None, cors_allow_origin: , watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }”},“target”:“text_generation_launcher”}
2023/12/14 13:51:44 ~ Error: DownloadError