Freeze weights of a zero shot model

Once deciding on a few constant labels, can one freeze the current weights of the models, to ensure a quicker inference time?