Supported models

🤗Optimum
1

Trying to compile a transformers model I’m using
Getting

KeyError: "hyenadna is not supported yet for transformers. Only ['...'] 
are supported for the library transformers. 
If you want to support hyenadna please propose a PR or open up an issue."

how/who decides which new models will be supported ?

2 Likes
2

Hi @ES-CONVERGE, what is the command/script you used to perform this conversion?

2 Likes
3

@ES-CONVERGE hyenadna uses remote code, it should be compatible with the APIs in Optimum in general, but as it’s not native citizen in the Transformers libraries. The best approach IMO would be creating a custom backend config and register it to the task manager of Optimum. WDYT @regisss?

3 Likes
4

@regisss
Tried running the following, note that model_dir Is just a local path to my model directory

optimum-cli export neuron \
--model ./model_dir \
--batch_size 128 \
--sequence_length 8000 \  
model_dir_neuron/
1 Like
5

@regisss @Jingya
worth mentioning I also tried with the torch_neuronx SDK on a inf2 EC2
Got the following error

2025-01-10 12:05:29.000929:  9138  ERROR ||NEURON_CC_WRAPPER||: Failed compilation with ['neuronx-cc', 'compile', '--framework=XLA', '/tmp/ec2-user/neuroncc_compile_workdir/d15d4161-30e4-4141-81b3-1028ce49c523/model.MODULE_15049036111489798686+e30acd3a.hlo_module.pb', '--output', '/tmp/ec2-user/neuroncc_compile_workdir/d15d4161-30e4-4141-81b3-1028ce49c523/model.MODULE_15049036111489798686+e30acd3a.neff', '--target=trn1', '--verbose=35']: Process Process-1:
Traceback (most recent call last):
  ...
neuronxcc.driver.Exceptions.CompilerInvalidInputException: ERROR: Failed command  /home/ec2-user/.local/lib/python3.9/site-packages/neuronxcc/starfish/bin/hlo2penguin --input /tmp/ec2-user/neuroncc_compile_workdir/d15d4161-30e4-4141-81b3-1028ce49c523/model.MODULE_15049036111489798686+e30acd3a.hlo_module.pb --out-dir ./ --output penguin.py --layers-per-module=1 --emit-tensor-level-dropout-ops --emit-tensor-level-rng-ops
------------ 
Reported stdout: 
DEBUG: needsModular? No. macCnt 0
INFO: Switching to single-module compile. PrePartitionPipe skipped.
INFO: Found memory bound graph
Replaced 0 dropout sequences with OffloadedDropout
INFO: HloMacCount has found 0
INFO: Traffic has found 49158152
INFO: AIF 0
HLO Ops used in computation: broadcast divide multiply parameter reshape tuple 
Invoking RemoveOptimizationBarriers pass
2025-01-10 12:05:28.050391: F hilo/MLIRPasses/Transforms/MhloToPythonPrinter.cc:732] ERROR: Unsupported type: complex<f32>

my code

import torch
import torch_neuronx
from transformers import AutoModelForSequenceClassification, AutoTokenizer


HF_TOKEN = "x"

model_id = "x/x"

print("Loading model + tokenizer")
tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
    token=HF_TOKEN
)

model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
    token=HF_TOKEN,
    num_labels=1
)
model.eval()

example_input = tokenizer("ACGTACGT", return_tensors="pt", truncation=True, padding="max_length", max_length=8000)["input_ids"]

print("Tracing the model with torch_neuronx...")
neuron_model = torch_neuronx.trace(model, example_input)

# Save the compiled model
neuron_model.save("x.pt")
print("Saved compiled neuron model: x.pt")
1 Like
6

Hi @ES-CONVERGE, I got the same compilation issue when exporting hyenadna, it seems that the tracing failed since the complex type complex <f32 > is not yet supported by the neuron compiler. I would suggest you to open a ticket in GitHub - aws-neuron/aws-neuron-sdk: Powering AWS purpose-built machine learning chips. Blazing fast and cost effective, natively integrated into PyTorch and TensorFlow and integrated with your favorite AWS services to report to the Annapurna team.

2 Likes
7

Thanks for the update - will do so

1 Like