I am a littile confused. In transformer based LLM inference. We always set the input length as the max_input length and padding 0 , or we dynamically get the actual input length and decrease the computation?

Inference input token number set as the max length always?

vivek9840 April 21, 2024, 8:28pm 11

these are the sources which i learned from. might help you.

Topic		Replies	Views
Sequences shorter than model's input window size 🤗Transformers	2	955	January 4, 2022
Confused about max_length and max_new_tokens 🤗Transformers	5	20127	December 14, 2023
Model max length not set. Default value 🤗Transformers	0	327	November 30, 2023
Change input_ids via API Inference for Text Generation Beginners	0	70	April 10, 2024
Limit max # of tokens for inference in pipeline? Beginners	0	827	April 7, 2023