Error code 400 when deploying huggingface bigscience/bloom to SageMaker

Beginners
1

Using the sample code below:

Screen Shot 2022-08-23 at 11.28.08 PM
Screen Shot 2022-08-23 at 11.28.08 PM1520×1556 295 KB

I’m getting the following error:

{
  "code": 400,
  "type": "InternalServerException",
  "message": "\u0027bloom\u0027"
}

Any ideas on what could be causing this issue?

1 Like
2

Having the same issue. Did you end up resolving it?

3

@mel-zheng it’s because BLOOM requires transformers version 4.21.0 but the inference containers offered only supports up to version 4.17.0. I ended up not using SageMaker. I went with a serverless approach and leverage our custom container that has transformers version 4.21.0 installed. Even tho I got the BLOOM model working, it is so large it is practically unusable. It’s unbearably slow running any inference.

4

Thanks for the update! I actually resolved it by supplying my own custom inference.py with the latest transformers version. Ended up using bloom-560m because bloom is too large like you said.

5

Hi @mel-zheng did you solve the issue?
Could you share your solution?