I have a use-case that seems like transformers should be able to solve, but I haven’t had great results just yet. Basically, if I have a bunch of reviews about a product (on the scale of 100-1000), I would like to be able to tell how many times a certain opinion is stated. For example, I’d like to know how many reviews say they liked feature A or that thought the price was reasonable.
My current approach is to use an nli model and pass a statement such as “Feature A is good” alongside the review. I would then get the probabilities of entailment, contradiction, and neutral to see if the model thought the review agreed/disagreed/had no opinion. I’ve found that this is barely better than randomly guessing. It is a tricky problem because there are so many nuanced ways that people can express support/criticism about a topic.
I don’t want to bother labeling data and using a supervised approach because I have too many opinions to check and those opinions change over time (e.g. right now we might be interested in knowing what people think about price, but later we will be interested in opinions on convenience). Maybe I am under the wrong impression about how many labeled data points are necessary, but I thought it would be impractical.
Has anyone else successfully built something similar? Which models or approaches did you use?