Hi Team, Whilst using the inference API to produce summaries of call using a private model I sometimes get different outputs to when I load the model and tokeniser locally even though I’m using the exact same parameters. To generate the summary locally I run: device = 'cuda' if cuda.is_available(…

Different Summary Outputs Locally vs API for the Same Text

philschmid December 3, 2021, 10:01am 4

pinging @Narsil since he is the master of the Inference-API and might know more.

1 Like