Accessing hidden states faster/only some hidden states

I am trying to access output of hidden layers for BLIP caption generator. I observed that this significantly slowed down the inference. it is taking approximately 6 seconds for inference. Is there any way I can acess only certain hidden states instead of outputs from all hidden states