Doubts in my project

Beginners
1

Hi there,

Myself Sharathsoorya T, studying Masters in Artificial Intelligence and Data Science. In my final year I need to do project and enrolled the title as Multimodal RAG based fashion Recommendatiom System. I am new/beginner in generative AI. If I did a project in recent blooming topics like RAG, I thought it would be beneficial for my job search. But I stuck while I started because I don’t know where to use LLM in my project. Fashion dataset will be stored as vectors in vector database and using some recommendation algorithm I will retrieve from the same vector database and give as output so here I can’t integrate LLM. At first I thought I can use LLM for recommendation but when I start to do the project I don’t know how to use LLM for recommendation. Without changing the meaning of title can I able to integrate RAG, LLM and recommendations. Anyone suggest how to do my project.

2 Likes
2

RAG allows for diverse configurations and typically centers around LLM and embedding models, so it should be possible to leverage LLMs while maintaining the core purpose of the title. If you haven’t learned Python yet, you should learn Python. If you know Python, I think starting with HF Learn is a good idea.

2 Likes
3

Thank you Mr. John for your valuable suggestion. I knew Python, the main problem of my project is LLM is used for generation, but I thought I can do recommendations with the help of LLM. Now I am struggling that I can’t use LLM for recommendations, so can we use LLM for recommendations or not. Thank you for sharing your work related to this, it will be a great help for me.

1 Like
4

I was studying for a postgraduate in linguistics till I got distracted by AI. I needed an AI to help me learn, so I had to teach it the syllabus. Sadly, LLM training does not include linguistics. I think (this was a year ago) I got a free GPT token, then I wrote a lot of training questions with multiple choice answers and put it in an Excel spreadsheet, then I trained it. It went from failing the DELTA to passing the DELTA. I had some success. You need a LOT of training questions… Google “finetuning”?

1 Like
5

can we use LLM for recommendations or not.

We can use them for that purpose. However, LLMs of the Instruct model, trained as a chatbot, is unpredictable and not well-suited for highly structured output. Therefore, when incorporating it into RAG workflows, some coding will likely be necessary to compensate for this.

If we can format the input and output of LLMs and/or embedding models to some extent in Python, they can interact effectively with other components. While system prompt exerts some coercive power, you shouldn’t expect strictness with LLMs…

# pip install -U transformers json_repair
import json, torch
from json_repair import repair_json
from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda" if torch.cuda.is_available() else "cpu"
MODEL = "Qwen/Qwen2.5-1.5B-Instruct"  # any chat model
tok = AutoTokenizer.from_pretrained(MODEL)
llm = AutoModelForCausalLM.from_pretrained(MODEL).to(device)

def _parse_json(s: str):
    a, b = s.find("{"), s.rfind("}")
    if a < 0 or b <= a: 
        return None
    blob = s[a:b+1].strip()
    try:
        return json.loads(blob)
    except json.JSONDecodeError:
        try:
            return json.loads(repair_json(blob))
        except Exception:
            return None

def llm_pick(request, items, top_n=5):
    catalog = "\n".join(
        f"{i+1}. {it['id']} | {it['title']} | {it.get('color','-')} | {it.get('fabric','-')} | {it.get('price','-')}"
        for i, it in enumerate(items)
    )
    system = (
        'You are a fashion recommender. Choose ONLY items from CATALOG.\n'
        'Output strictly this JSON:\n'
        '{"chosen_ids":[string], "reasons":{"<id>":"one short reason using color/fabric/price or match to request"}}\n'
        'Use double quotes. No prose. No code fences.'
    )
    user = f"REQUEST: {request}\nN_TARGET: {top_n}\nCATALOG:\n{catalog}"

    msgs = [{"role": "system", "content": system}, {"role": "user", "content": user}]
    inputs = tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(device)

    out = llm.generate(**inputs, max_new_tokens=512)
    text = tok.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]

    obj = _parse_json(text)
    if obj is None: # to avoid crashing
        mentioned = [it["id"] for it in items if it["id"] in text][:top_n]
        obj = {"chosen_ids": mentioned, "reasons": {}}
    return obj

items = [
    {"id":"dress01","title":"Red floral midi dress","color":"red","fabric":"cotton","price":1499},
    {"id":"shirt01","title":"Beige linen short-sleeve shirt","color":"beige","fabric":"linen","price":1299},
    {"id":"sneaker01","title":"White low-top sneakers","color":"white","fabric":"leather","price":2499},
    {"id":"jeans01","title":"Blue straight-fit jeans","color":"blue","fabric":"denim","price":1999},
]
print(llm_pick("beach wedding outfit under 2000 rupees, breathable", items, top_n=2))
#{'chosen_ids': ['dress01'], 'reasons': {'dress01': 'Red floral midi dress is both affordable and breathable for a beach wedding.'}}
print(llm_pick("light summer outfit under 3000 rupees", items, top_n=2))
#{'chosen_ids': ['dress01', 'jeans01'], 'reasons': {'dress01': 'Light and airy, perfect for summer', 'jeans01': 'Comfortable and affordable, suitable for a casual look'}}
1 Like