Git LFS fail for Spaces with Github Workflow

I face the same issue as well, I had previous commits to binary files without lfs commits but later on, I enabled git lfs but still my workflow fails since the file is there in the previous commits. How do I fix this?