Using model.generate() in parrellel / faster?

jared312 · October 11, 2023, 3:45am

Hey all,
I have fine tuned a llama model and have loaded it into a model object with AutoPeftModelForCausalLM.

I now want to used this model for inference on a lot of queries. Currently I am looping (yes for looping) through each query and calling mymodel.generate, then tokenizer.decode to get each response.

Is there a smarter way I can do this in huggingface? Can you generate in parrellel or batches? I have 4 GPUs avaliable to me.

Thanks
Jared

Topic		Replies	Views
Why is the huggingface generater much slower than the original llama2 generater? 🤗Transformers	0	555	November 23, 2023
Using the .generate() function with a custom model class Models	0	503	March 3, 2023
Hugging Face Llama-2 (7b) taking too much time while inferencing Models	0	912	November 13, 2023
Using huggingface generate() with custom model 🤗Transformers	1	1035	August 26, 2023
Custom model.generate() parameters for hosted models Beginners	0	293	October 7, 2021

Using model.generate() in parrellel / faster?

Related Topics