Not able to run on DML with pipeline

🤗Optimum
1

Hi Experts,

I’m new to HuggingFace/Optimum, in fact I started working on AI/ML stuff recently.

I tried to optimum models with DML EP (on my windows PC), for example take optimum/vit-base-patch16-224 · Hugging Face

model = ORTModelForImageClassification.from_pretrained(model_name, provider=“DmlExecutionProvider”)

Initially nodes are distributed between CPU EP & DML EP, but after “pipeline” all those nodes were placed on CPU EP

Why the pipeline is overwriting the execution provider as CPU EP? I want to run nodes on DML also, how can I do that with pipeline

Thanks

2

Hi ! it seems this is because in transformers pipeline, the model is moved to the model’s device (cpu in this case) here and the .to() method in optimum doesn’t consider this provider when doing so.
This can be fixed by adding a catch (if device==cpu and provider==DmlExecutionProvider) the same way we do for TensorrtExecutionProvider here
Please try this fix and we’d be glad if you would like to open a PR in Optimum to solve it.

2 Likes
3

Hi @IlyasMoutawwakil

Thanks for your suggestion. I see this issue is resolved with the latest packages.

onnx 1.16.1
onnxruntime 1.18.0
onnxruntime-directml 1.18.0
optimum 1.20.0

Now “pipeline” is not effecting the nodes distribution. I see nodes got distributed b/w CPU and DML EP’s

Thanks