Bertweet pooler_output for random individual words is almost identical...why?

Models
1

Out of curiosity I am looking at the pooler_output for individual words (like cat) using Bertweet:

model = AutoModel.from_pretrained("vinai/bertweet-base")
tokenizer = BertweetTokenizer.from_pretrained("vinai/bertweet-base", use_fast=True)
text = "cat"
normalized = tokenizer.normalizeTweet(text)
input_ids = torch.tensor([tokenizer.encode(normalized)])
outputs = model(input_ids)['pooler_output'].numpy().squeeze()

I did this for several words (“cat”, “dog”, “god”, “truck”, “rooster”) and plotted a sampling of the outputs (every 10th point):
download (3)

I was a bit surprised that the vectors would be so similar. I’m assuming that the way the model is built there is some “baseline” feature when only a single word is input and that is what is being seen here mostly. Is there a good explanation somewhere of this phenomenon?