I am trying to do the static quantization on the T5 model(flexudy/t5-small-wav2vec2-grammar-fixer) for reducing the inference time . Code : import torch import transformers from transformers import T5Tokenizer, T5ForConditionalGeneration model_name = “flexudy/t5-small-wav2vec2-gram…

Model quantization

Hemanth-thunder June 7, 2022, 7:01am 2

use dynamic quan.

Topic		Replies	Views
Error saving quantized model Intermediate	4	3948	February 16, 2023
T5 inference performance Models	5	1567	March 8, 2022
ValueError: The model is quantized with QuantizationMethod.QUANTO and is not serializable 🤗Transformers	1	333	May 20, 2024
Boost inference speed of T5 models up to 5X & reduce the model size by 3X 🤗Transformers	2	5622	June 8, 2023
Quantization of facebook/opt-13b model 🤗Transformers	0	1006	July 28, 2022