I tried bert and bart to output embedding of document and query.
Bart is much better than bert ,.
However in zero-shot learning condition, the embedding of sentence looks good when they have similar length, but when using embedding of query which is really short, the embedding is much worse. The embedding of query is much closer to the short rubbish data other than the long relative data.
Are there good models or training tasks to map query and document into a similar space which has no relation with the length of sentences?
Have you read this article? There is a small portion that discusses why models that produce sentence embeddings are bad at embedding one or two words into a relevant latent space.
Nice article.
I tried using fully connected layers to map embedding of queries, but the query has variable length other than the single word label which made it much more complicated and having bad result.
Right here are two options off the top of my head.
- Take the average of all of the output embeddings
- Use the CLS embedding (if it is a BERT-ish model)
This will ensure that you always have the same vector size (768, I think)