[Release] Avalanche: Ultra-Low VRAM Matrix-Factorized Optimizer for PyTorch (Up to 99.99% Memory Reduction)

[Release] Avalanche: Ultra-Low VRAM Matrix-Factorized Optimizer for PyTorch (Up to 99.99% Memory Reduction)

Hey everyone :waving_hand:

At OpenBlade, we just open-sourced Avalanche, a standalone, ultra-low VRAM matrix-factorized optimizer designed to break the memory wall in foundation model pretraining and fine-tuning.

The Problem: The Optimizer Memory Wall

In deep model training, optimizer states frequently consume significantly more VRAM than the model weights themselves.

Standard FP32 AdamW requires storing two states per parameter—the first momentum vector m and the second uncentered variance vector $v$—costing 8 bytes of VRAM per parameter:

  • For a 1.1B parameter model, standard AdamW eats ~8.96 GB solely in optimizer states.
  • For a 70B model, it demands ~560 GB just for optimizer bookkeeping before loading a single token or activation.

The Solution: From Area O(M \\cdot N) to Perimeter O(M + N)

Instead of storing an M \\times N dense grid of running squared gradients, Avalanche shifts second-moment estimation to a Rank-1 marginal factorization:

\\hat{v}\_{ij} = \\frac{r_i \\cdot c_j}{g_m \\cdot (1 - \\beta_2^t)}

where row marginals r \\in \\mathbb{R}^M and column marginals c \\in \\mathbb{R}^N track mean squared gradients across rows and columns.

  • For a 4096 \\times 4096 projection layer, state size drops from 16.7M floats (67.1 MB) down to 8,192 floats (32.8 KB)—a 99.95% reduction per linear layer.
  • In zero-momentum mode, the internal state sits under a flat ~10 MB ceiling across foundation models, freeing over 550 GB of VRAM on a 70B architecture.

Memory Benchmarks Across Model Scales

Model Scale Parameters Standard FP32 AdamW (m+vm + vm+v) 8-bit Adam (bitsandbytes) Avalanche State Net Reduction
Small / Edge 125M ~1.00 GB ~250 MB ~1.8 MB 99.82%
Foundation 1.12B ~8.96 GB ~2.24 GB ~9.6 MB 99.89%
7B Base 7.00B ~56.00 GB ~14.00 GB ~24.5 MB 99.96%
13B Scale 13.00B ~104.00 GB ~26.00 GB ~42.0 MB 99.96%
70B Scale 70.00B ~560.00 GB ~140.00 GB ~78.0 MB 99.99%

Multi-Precision Velocity & Lossless Error Feedback

Avalanche supports multiple first-moment momentum modes depending on your available VRAM headroom while keeping second moments factorized:

  • Avalanche (mode="none"): Pure zero-momentum with dynamic RMS normalization. 0.0 B / param momentum overhead and a flat ~10 MB memory ceiling.
  • Avalanche4Bit (mode="4bit"): Packed 4-bit signed nibbles (0.5 B / param) with lossless Error-Feedback (EF) compensation.
  • Avalanche8Bit (mode="8bit"): Signed 8-bit integers (1.0 B / param) with EF compensation.
  • AvalancheBF16 / AvalancheFP16: Native half-precision velocity (2.0 B / param).
  • AvalancheFP32: Full single-precision float velocity with factorized second moments (4.0 B / param).

Lossless Error-Feedback (EF)

In quantized modes (4bit, 8bit), quantization residuals are stored in compact half-precision buffers (e_t = m_t - \\hat{m}\_t) and injected back into the next momentum step:

m\_{t+1} = \\beta_1 (\\hat{m}\_t + e_t) + (1 - \\beta_1) G\_{t+1}

Because residuals accumulate over time, no gradient signal or update magnitude is discarded.

Key Architectural Highlights

  • Dynamic RMS Normalization: Automatically bounds update magnitudes (u \\leftarrow u / \\max(1.0, \\text{RMS}(u))) to prevent gradient explosions in BF16/FP16 without requiring aggressive gradient clipping.
  • Chunked Streaming for Large Tensors: Slices massive matrices (like 65k embedding heads) into row tiles to eliminate transient VRAM spikes during update reconstruction.
  • Zero External Dependencies: Built natively on PyTorch with no custom CUDA compilation or external C-bindings required.

Quickstart

Avalanche is a drop-in replacement inheriting directly from torch.optim.Optimizer:

Python

import torch
import torch.nn as nn
from avalanche import Avalanche, Avalanche4Bit

model = nn.Transformer(d_model=1024, nhead=16).cuda()

# Option A: Zero-momentum flat ceiling (~10 MB state across any scale)
optimizer = Avalanche(model.parameters(), lr=1e-4, weight_decay=0.01)

# Option B: Lossless 4-bit momentum with error-feedback
optimizer = Avalanche4Bit(model.parameters(), lr=1e-4, betas=(0.9, 0.999), lossless=True)

for x, y in dataloader:
    optimizer.zero_grad(set_to_none=True)
    loss = criterion(model(x), y)
    loss.backward()
    optimizer.step()

Integration with Hugging Face Trainer

Python

from transformers import Trainer
from avalanche import Avalanche

class CustomTrainer(Trainer):
    def create_optimizer(self):
        self.optimizer = Avalanche(
            self.model.parameters(),
            lr=self.args.learning_rate,
            weight_decay=self.args.weight_decay,
            betas=(0.0, 0.999),
        )
        return self.optimizer

Getting Started

You can install it via pip or drop optimizer.py directly into your repository:

Bash

pip install git+https://github.com/OpenBladeAI/Avalanche.git

Give it a run on your setups or clusters and share your training loss curves with us here on the forum!

2 Likes