PEFT vram usage


So im trying out peft and seemingly vram usage is excessive, for instance i am executing the example on “bigscience/bloomz-3b” with the provided accelerate configuration accelerate_ds_zero3_cpu_offload_config.yaml on a system with 3x AMD Radeon Instinct MI50 accelerators with 16GB vram eatch like this:

accelerate launch --num_processes=3 --config_file=accelerate_ds_zero3_cpu_offload_config.yaml ./

this very quickly fills the vram of all three cards and then ooms

I have also written some custom training scripts that run into the same problem.

This feels very wrong. Im unsure how to debug this/ profile the vram usage of the peft/transformers stack.