Combine Text Embedding from LLM's with other features for similarity search

Hello,

Item 1 = Text features, latitude, longitude, age of each item, language, number of visits, number of clicks …
Item 2 = Text features, latitude, longitude, age of each item, language, number of visits, number of clicks …

I built a recommendation system based on cosine similarity of llm embeddings for text features

How do I combine text embeddings with other features, and use the combination for cosine similarity ?

A simple approach = concatenate one hot encoding of categorical features with text embeddings from llm. This does not seem right since the one hot encoding, and text embeddings are representations from different vector spaces.

Looking for suggestions