Hi, I have an image tensor of shape (1, 3, 224, 224) and I want to get its embedding from the
CLIPVisionModel with the
Now when I pass the tensor to the vision model, I get two outputs with the following shapes:
last_hidden_state: [1, 50, 768]
pooler_output: [1, 768]
In one discussion thread, I read that choosing
last_hidden_state[:, 0] represents the overall representation of the image while in another thread I read choosing
pooler_output will be better. I’m bit confused about this, can anyone please help me with this?
Yes there are typically 2 ways to get a “pooled” representation of an entire image. One is taking the
last_hidden_state and average them across the sequence dimension. So you could do
last_hidden_state.mean(dim=1) and use this as your image representation.
An alternative representation is indeed the
pooler_output, which takes the embedding of the first special CLS token from the
last_hidden_state, and applies a layernorm to it as seen here.
I’d say that average pooling all last hidden states vs. taking the CLS token’s representation give similar results. However the original ViT authors actually released a new paper in which they replaced the use of CLS token by average pooling and they got better results.