Good day experts
I need to build a system to recommend related rows.
If query = row1, I need results ranked by similarity which combines signal from all attributes.
Each row has multiple text attributes.
Each of these attribute is a independent document.
Attribute1 = doc1, Attribute2 = doc2… and so on.
Number of documents per row is variable
Row 1 may contain Attribute1 = doc1, Attribute 2 = doc2.
Row 2 may contain Attribute1 = doc1, Attribute 2 = doc2 … Attribute10 = doc10.
I considered two approaches.
- Concatenate first n characters of each attribute (doc) and get embeddings for one concatenated string
- Get embeddings for each attribute (doc) separately
Approach 1 saves us cost of fetching and storing embeddings. This requires concatenating chunks from separate documents. I have a concern. This approach may result in a concatenated string with abrupt change in content and semantic meaning at the points of concatenation.
Approach 2 is costly, we need to fetch embeddings for multiple documents per row. However I think, the embeddings are more usable as each document retains its meaning.
Question 1 - Which approach do experts recommend 1 or 2, and why ?
Question 2 - Any guidance on doing similarity search for embeddings from multiple text attributes ?
Can we simply concatenate the embeddings ?
How do we handle rows with different number of attributes ie., different length of concatenated embeddings ?