"Workload Evicted, Storage Limit Exceeded (50G)" when swap is merely at 16G

You’ve probably seen this topic a lot but I am trying to get to the bottom of this. My Docker image is 8G in size and my service does not download any data or write to disk during runtime. So I assumed that the reason for why the storage limit is exceeding 50G is because my server is somehow causing the space to swap out more than 50G. So I decided to monitor this metric.

In the graph above, we can see that virtual memory usage peaks at 25G, while physical memory usage stays below 5G. However, my space evicts the workload at 07:21 A.M. when virtual memory usage is only at 16G, and if my math is right, 16G + 8G = 24G, which is nowhere near 50G.

My only other explanation for this is that HF Spaces is running another process in the background that is also swapping to disk along with my own service.

1 Like

When in use, cache and other data often consume disk space on Spaces.


Your Space was evicted because the ephemeral root disk hit the 50 GB cap, not because of swap. Free CPU Spaces provide 16 GB RAM, 2 vCPU, and 50 GB non-persistent disk. Crossing that disk quota evicts the workload, regardless of RAM or swap graphs. (Hugging Face)

What actually counts toward the 50 GB

Everything written to the container’s filesystem: image layers, the container’s writable layer, /tmp, app and system logs, package caches, HF caches, and any swapfile. Docker stores images as multiple layers plus a writable layer that grows with your app’s writes. The writable layer is separate from memory usage and easily expands past what the “image size” suggests. (Docker Documentation)

Why your memory math doesn’t match disk evictions

Memory graphs measure RAM and swap, not root disk. An 8 GB image + 16 GB swap says nothing about additional writes to the writable layer, caches, and logs. OverlayFS layers and the writable layer can consume far more disk than you expect, and logs can silently grow. (Docker Documentation)

Common culprits on HF Spaces

  • HF caches default under ~/.cache/huggingface/... unless redirected. This includes huggingface_hub, transformers, diffusers, and datasets. (Hugging Face)
  • /tmp and framework intermediates.
  • Logs accumulating in the container or host JSON logs. (Docker Community Forums)
  • Layer bloat from package caches and build debris that increase the writable layer. (Docker Documentation)

How to confirm the real disk usage

Run these inside the container to see what’s filling disk:

# root filesystem usage
df -h /

# biggest top-level dirs on this filesystem
du -xhd1 / 2>/dev/null | sort -h

# usual hotspots
du -xhd1 /root /home /usr /var /tmp 2>/dev/null | sort -h
du -xsh ~/.cache/huggingface ~/.cache 2>/dev/null || true

# large single files
find / -xdev -type f -size +1G -not -path "/proc/*" -printf "%p %s\n" 2>/dev/null

Fixes and durable workarounds

1) Move HF caches off the ephemeral disk

Attach Persistent Storage and redirect caches to /data so they don’t count against the 50 GB root cap.

# HF caches to persistent disk
# Docs: https://huggingface.co/docs/hub/en/spaces-storage
ENV HF_HOME=/data/.huggingface \
    HF_HUB_CACHE=/data/.cache/huggingface/hub \
    TRANSFORMERS_CACHE=/data/.cache/huggingface/transformers \
    DIFFUSERS_CACHE=/data/.cache/huggingface/diffusers \
    HF_DATASETS_CACHE=/data/.cache/huggingface/datasets

Spaces docs show /data and HF_HOME=/data/.huggingface; the hub cache can be moved with HF_HOME or HF_HUB_CACHE. (Hugging Face)

2) Clean existing caches

Prune old hub cache entries before restart:

# CLI documented in the Hub cache guide
hf cache prune -y

(Hugging Face)

3) Keep /tmp and logs small

Set TMPDIR to /data/tmp if you have persistent storage, and emit application logs to stdout with rotation or reduced verbosity. Large logs are a frequent cause of overlay growth. (Docker Community Forums)

4) Build hygiene to reduce layer bloat

Use slim bases and multi-stage builds. Delete package caches in the same RUN step. Example:

RUN apt-get update && apt-get install -y --no-install-recommends git \
 && rm -rf /var/lib/apt/lists/*    # apt lists removed

RUN pip install --no-cache-dir -r requirements.txt  # avoid pip cache

These are standard Docker best practices for space efficiency. (Docker Documentation)

5) Avoid downloading whole datasets

Use streaming to iterate without fully storing datasets on disk:

from datasets import load_dataset
ds = load_dataset("c4", "en", streaming=True)

Streaming is designed for low-disk environments. (Hugging Face)

6) If you truly need more disk

Add Persistent Storage and keep large artifacts under /data. The ephemeral root disk remains 50 GB on CPU tiers. (Hugging Face)

Context you can rely on

  • Free CPU Spaces: 50 GB non-persistent disk. Eviction occurs when the container’s root filesystem exceeds that. (Hugging Face)
  • HF provides official guidance to place caches under /data via HF_HOME. (Hugging Face)
  • Docker’s writable layer grows with your runtime writes; this is independent of memory use. (Docker Documentation)
  • Logs and overlay2 behavior are common reasons for unexpected disk growth. (Docker Community Forums)

References and quick reads

I appreciate the response but my container doesn’t write to disk during runtime, except for 20 bytes when you first load the Swagger page.

1 Like

About a month ago also I noticed this too issue too, in the case of Annif - Text Classification and Subject Indexing - a Hugging Face Space by NatLibFi. With a free account I cannot ssh to the instance to debug.

Any advice, anyone?

1 Like

I have started monitoring the disk usage for the last 12 hours, and as expected, it’s stable at < 200MB

2 Likes

Did you solve it? If yes, how?

I’m facing the same issue and I can’t understand what’s causing the storage to overload

1 Like

No, I haven’t been able to solve it, and I am pretty sure it’s unsolvable. Here’s my disk usage for the last few week. It never once exceeded 300MB, yet I still had the error a couple of times.

FWIW, my friend created a new Hugging Face account and deploys his services on multiple spaces, and routes incoming requests to those spaces using a load balancer.

1 Like