Gave up and tried disabling LFS, now I get this error when trying to push:
Enumerating objects: 24029, done.
Counting objects: 100% (24029/24029), done.
Delta compression using up to 16 threads
Compressing objects: 100% (24027/24027), done.
error: RPC failed; HTTP 408 curl 22 The requested URL returned error: 408
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (24028/24028), 7.00 GiB | 20.13 MiB/s, done.
Total 24028 (delta 0), reused 24028 (delta 0), pack-reused 0
fatal: the remote end hung up unexpectedly
Everything up-to-date