ViT Model increasing CPU RAM when moving to GPU

Hi,

Unlike with a classical torch.nn.Module, moving a freshly initialized ViTModel to GPU increases CPU RAM by the same amount.
Anyone know a fix?