Hi,
Yes there are typically 2 ways to get a “pooled” representation of an entire image. One is taking the last_hidden_state and average them across the sequence dimension. So you could do last_hidden_state.mean(dim=1) and use this as your image representation.
An alternative representation is indeed the pooler_output, which takes the embedding of the first special CLS token from the last_hidden_state, and applies a layernorm to it as seen here.
I’d say that average pooling all last hidden states vs. taking the CLS token’s representation give similar results. However the original ViT authors actually released a new paper in which they replaced the use of CLS token by average pooling and they got better results.