Unfortunately, soon after I wrote that, I realized beam search (.generate()) was actually not working with EI (though .forward() does, but using that means losing out on beam search). I faced the same situation whereby the model inference was taking place on CPU only, no EI, so had to eventually switch up to a GPU instance.
That was ~2 months ago now, so don’t know if something has changed since I haven’t looked back.
A couple other things:
- As of writing (and this has been the same for a few months now) EI on PyTorch is only supported on PyTorch 1.3.1. So, if using any other version of PyTorch, it will fallback to CPU.
- Since PyTorch 1.6 the way in which the models are saved has changed. To make a model backward compatible, it might have to be saved with
_use_new_zipfile_serialization=False.
Sorry I couldn’t help more. Please do share if you find other workarounds here. Thanks!