AI Halucinations, User AI Surfing, and AI Ratings

Beginners
1

So I’ve been thinking about something lately, and I’m wondering what all your opinions might be on the subject. There are obviously different models that are offered for public use for various tasks. If we look at some of the big ones like Gemini, ChatGPT, GROK, Claude, how do the companies that host them rank whether or not a model is overall valuable?

Now that’s an interesting question in and of itself, but let’s look at something else.

I got to watch this firsthand with my business partner. I’ve also seen quite a few videos on it. One of the videos is pretty good: ChatGPT gets a basic physics question wrong. ChatGPT was not inherently asked to solve the physics. Instead, it was asked to observe a visual field (a video feed of a user holding a pencil at both ends). The user explained that he was going to let go of one end of the pencil and wanted ChatGPT to tell him what was happening. And ChatGPT very confidently stated that the pencil would fall to the ground.

The interesting thing is that the video concludes the video concludes that ChatGPT hallucinates. I do not know if the user was trying to build a bigger point out of that, but I think the point he made about it hallucinating is enough for my point. How often do new users or even intermediate users try to find out why an AI hallucinates? We don’t really know much about how an AI generates answers, even though we use the term loosely. It’s interesting how semantic Cloude in AI works, how they build statements and things out of probability. In this example, I actually know what happened.

The AIs are mostly trained on text information, so video feed information has a lower priority. That means ChatGPT likely wasn’t specifically observing the video feed. It was more likely relying on pre-trained answers.

Is it very likely the user in this case would have intuited that, or even searched for it? Does the average user feel particularly motivated to do that? Or are they going to do what I’ve seen in quite a few threads: just surf from model to model, using the model as long as it looks accurate and like it’s doing great, until it starts hallucinating?

I wrote a series of papers, and I posted one here about prompt engineering. Prompt engineering is an interesting area of research. It doesn’t seem to me like it’s being pushed as much as it was before. I’m not sure anybody is focusing on the end user as much as they seem to be focusing on teaching researchers how to use AI.

In Watching my business partner work, I realized something. He had been working with AI for several years and had studied prompt engineering. As soon as a more powerful model became available on a completely different platform, he would drop one AI and switch to it, often using it for only a week or so before moving on to another one because it hallucinated, got basic things wrong, etc.

When I personally observed them in our shared office, and in the few times we worked directly together in the same location, I noticed that all of the AIs were making the same basic mistakes. And notice I’m not talking about different models under the same brand, like different Gemini models. I’m talking about wide gambits of different entire companies and platforms. This made me wonder how many people actually noticed that. How many end users using AI actually notice this?

I also wonder about the volume of end users versus people who are actually working with, researching, building, and training AIs. How many end users have no idea what the AI even is and think it’s just an advanced chatbot? How many people are actually researching, working on, and building AI, and for all intents and purposes know a great deal about AI?

Why is any of this even important? If you think about it logically, if the end user (the people) don’t know much about AI to start with (maybe they researched it, maybe they didn’t), and they vastly outnumber researchers and builders, does that mean they get less weight in evaluations of AI when they choose to rate different AI models? If their volume gives them more weight, why would I want to consider that angle?

The question I asked earlier: What is it that different platforms use to rank their AI models?
Why is that important? I have noticed a bias in AI. I’m sure I’m not the only one who has noticed this. Different AI models tend to be very short-term task-satisfaction weighted, meaning they tend to take the shortest possible path to satisfy the user’s inquiry.

For simple questions, this may be fine, especially for long conversations. For trying to build complex programs, it can be pretty bad. We end up constantly asking why an AI trained on very high-level programming language and very architecturally sound programs builds such bad programming sets and codes so poorly. And don’t get me wrong, many times the code works. It just runs poorly. And yes I know models are getting better at this but that’s not the point I’m looking at.

What I’m observing is where these two things coincide: are major platforms getting downvoted on AI hallucinations by users who may or may not really understand how the AI works? Is that information feeding back into the platform, reinforcing the overall perspective that the AI model they’ve deployed may not be all that good?

Think about this: if you’re working with an AI that is a large language model, a large part of its content is generated through the user interacting with the context window, basically having a conversation. How much of the A.I.'s hallucination is driven by miscommunication and how much of that hallucination is generated by the ether, the unknown, the void? Is there anything in the conversation that can be managed by the user through conversation if conversation is indeed the main form of information transfer? How much hallucination cannot be managed by conversation?

And do end users know this?

I’m going to go out on a limb and say that if the overwhelming industry byline is that "AI is amazing, AI is the future, AI is awesome, everybody needs AI, and that the AI just knows what you want… " That the end user probably does not know this.

So what do you think, community? What are your thoughts on this overall?

Do you think end-users, who may or may not have a huge understanding of AI, can be a driving force in telling a platform whether its models overall are very good and able to satisfy user requests? If that is indeed the case, could there be downstream consequences for users from the huge gap between what they expect and what the model can actually do?

Because make no mistake, even in highly trained models if they are used poorly or interacted with poorly, they can render poor results.

2

Hello!

From my own testing, it sometimes feels like some models are optimized more for benchmark performance than real world usability. For example, Deepseek looks very strong on benchmarks, but in my personal use, it didn’t always perform as well as expected.

On the other hand, some models seem more optimized for everyday user interaction. In my experience, ChatGPT handles messy or noisy prompts quite well, while models like Qwen tend to require cleaner, more structured prompts. This suggests models are being optimized in different directions, some for high benchmark scores, others for practical user experience.

Regarding the physics mistake you mentioned, AI generally only “knows” what was in its training data. But it isn’t just about the data, size matters too. A small model, even with perfect data, often can’t handle long prompts or multi-instruction requests simply because of mathematical and architectural limitations. If a task requires more “reasoning” steps than the model’s parameters can support, or if the model has seen too little relevant data during training, you get those confident hallucinations you described.

3

Are we talking about the free or the paid tiers?

4

Hmm… even just framing this properly already looks pretty hard.:thinking:

My short answer: yes, hallucinations, user ratings, and “AI surfing” are interesting, but the deeper issue is that consumer AI ratings are not simply model ratings.

A normal user is usually not evaluating “GPT,” “Claude,” or “Gemini” as raw models. They are evaluating ChatGPT, Claude, Gemini, or another AI assistant as a consumer service.

That service can include a base model, routing, memory, search, tools, UI defaults, safety behavior, subscription limits, and fallback behavior. So “ChatGPT failed” can be a valid consumer statement while still being technically underspecified.

A simple analogy: if a car performs badly, the driver reasonably says “the car failed.” But an engineer still asks whether the problem was the engine, transmission, tires, software, fuel, road condition, driver interface, or maintenance state. Consumer AI has a similar attribution problem.

1. “ChatGPT failed” is a product observation, not necessarily a clean model observation

A user-visible failure may come from several layers:

Layer Why it matters Example
Model Base capability GPT / Claude / Gemini
Router Chooses fast, reasoning, tool, or fallback paths GPT-5-style routing
Memory Personalizes responses and changes reproducibility saved memories, chat history
Search / RAG Adds freshness, but can also add noise web search, retrieved context
Tools / agents Changes task execution, not just answer generation code, files, browsing
Policy / UI Shapes refusals, tone, visibility, and control safety behavior, model picker
Pricing / limits Changes perceived value and fallback behavior free vs paid tiers, caps

This is not just theoretical. OpenAI describes GPT-5 as a unified system with a smart/efficient model, a deeper reasoning model, and a real-time router that chooses a path based on conversation type, complexity, tool needs, and user intent (OpenAI GPT-5, GPT-5 System Card). ChatGPT can also use web search automatically or manually (ChatGPT Search, Search help). Memory adds another variable: OpenAI distinguishes saved memories from chat-history reference (Memory FAQ, What is Memory?).

So ordinary ChatGPT use is often not the same as sending a stateless prompt to a fixed model under fixed conditions.

The point is not that users understand all these layers. The point is the opposite: most users do not. They experience the stack as “ChatGPT,” “Claude,” or “Gemini.” From the user’s perspective, attribution rarely matters. The product failed. From an evaluation perspective, attribution matters a lot.

2. Search is not just a boost; it is an evaluation condition

Search is often treated as if it simply makes the system stronger. Sometimes it does. But search also changes the task.

A search-enabled assistant may be doing all of this: deciding whether to search, forming a query, retrieving sources, selecting evidence, compressing context, handling conflicts, integrating external information, and deciding how much uncertainty to show.

That can improve freshness. It can also introduce noise.

The AI Consumer Index / ACE is useful because it evaluates consumer tasks in shopping, food, gaming, and DIY with web search turned on. Even with search enabled, the top model scored 56.1%, and shopping remained especially difficult, with the top model under 50%. The paper also notes failures around prices and working links.

This matters because many consumer tasks are not just “know the fact.” They involve current prices, valid links, constraints, source quality, and real-world context.

RAG research points in the same direction: irrelevant retrieved context can reduce accuracy or mislead reasoning (RAG robustness paper). Long context can also hurt when relevant information appears in the middle of a large context window (Lost in the Middle).

So “was search on?” is not a minor implementation detail. It is a major evaluation variable. Search off, search manually on, search automatically triggered, search polluted by bad results, and search overused in a task that did not need it are different conditions.

3. “Performance” has several meanings

A lot of confusion comes from using one word, “performance,” for different things:

  • model performance: what the model can do in isolation
  • system performance: what the routed/search/memory/tool-enabled product does
  • interaction performance: what the user and assistant accomplish together
  • UX performance: whether the interaction feels useful, controllable, trustworthy, and recoverable
  • market performance: whether users continue, switch, pay, recommend, or complain

These are related, but not identical.

ChatBench argues that standard benchmarks such as MMLU measure “AI-alone” capability, while actual chatbot use is often human-AI interaction. It reports that AI-alone accuracy fails to predict user-AI accuracy across subjects such as math, physics, and moral reasoning.

UXBench evaluates AI assistants using user-experience signals from over 70K interaction logs. Beyond Benchmarks surveys active users across ChatGPT, Claude, Gemini, DeepSeek, Grok, Mistral, and Llama; it reports statistically similar satisfaction among the top platforms, more than 80% of users using two or more platforms, and hallucination plus content filtering as common frustrations.

That suggests the market may not simply converge on “the best benchmark model.” Users may multi-home or switch based on task, tone, trust, policy, interface, price, or a salient failure.

4. Consumer value is not benchmark performance

Users do not convert benchmark scores into value mechanically.

Classical technology-adoption and service research already gives useful tools here: UTAUT2 for usefulness, enjoyment, price value, and habit; Expectation-Confirmation Theory for satisfaction and continuance; SERVQUAL for gaps between expected and perceived service; PERVAL for multidimensional consumer value; and service-dominant logic for the idea that value is realized in use, not merely embedded in the product (Vargo & Lusch).

That maps well onto consumer AI:

Value type What users may care about AI example
Functional Accuracy, usefulness, speed “Did it solve my task?”
Emotional Tone, warmth, comfort “It feels colder now.”
Price Subscription value “Useful, but not worth paying.”
Familiarity Learned workflows and failure modes “I knew how to use the old one.”
Control Ability to choose settings model/search/memory toggles
Trust Transparency and recovery citations, uncertainty, correction
Context-fit Value in the user’s actual situation “Great benchmark, bad for my workflow.”

This is the key marketing problem: better is not a scalar.

In consumer AI, “better” may mean more accurate, faster, warmer, cheaper, more familiar, more controllable, more transparent, less filtered, less surprising, or simply less annoying.

A company may think:

“We made it smarter, safer, more personalized, and more automatic.”

A user may experience:

“It became colder, less predictable, more expensive, and harder to control.”

Both can be true.

5. Familiarity is a real source of value

Familiarity is easy to underestimate.

In ordinary software, users become familiar with layouts, shortcuts, workflows, and error patterns. In AI services, users also become familiar with conversational behavior: how much prompting the assistant needs, what tone it uses, what it tends to misunderstand, when it refuses, when it hallucinates, when search helps, when search hurts, whether it feels warm or rigid, and which tasks it is “good enough” for.

So a model or product update can destroy learned user competence even if aggregate benchmarks improve.

This is one reason old-model removal or automatic model switching can feel disruptive. The user may not only be losing a model. They may be losing a familiar working environment.

That is especially important for conversational AI because the “interface” is not only buttons and menus. The interface is also the assistant’s behavior.

6. Information asymmetry is unusually severe

Consumer AI should not be analyzed as if users were perfectly informed benchmark readers.

They are boundedly rational consumers facing opaque, frequently changing, personalized service stacks. In practice, users often rely on brand, first impressions, recent failures, social proof, price, habit, and “it works for me” heuristics.

This is not irrational in a simple sense. It is a reasonable response to limited information.

AI markets also have a strong information-asymmetry problem. A system can appear fluent, confident, and high-quality while hiding defects that only appear in specific domains, workflows, or edge cases. One recent paper explicitly frames AI consumer markets through a “market for lemons” lens: When Life Gives You AI, Will You Turn It Into A Market for Lemons?.

Memory makes the opacity sharper. A study of ChatGPT Memory describes an “algorithmic self-portrait” formed from user-disclosed information and notes concerns around opacity, agency, and sensitive inferences: The Algorithmic Self-Portrait.

So the consumer is not choosing among transparent machines. The consumer is choosing among branded, changing, partially hidden systems.

7. Failure types have different market effects

Not all failures are equal.

Failure type Likely user interpretation Market effect
Small factual error Annoying but recoverable Low to medium
Confident hallucination “I cannot trust this.” High
Bad citation or broken source “Bad for research.” High
Wrong visual/video interpretation “It cannot see what I see.” High
Over-refusal “It is useless / over-filtered.” Medium to high
Cold tone shift “The old one felt better.” Medium
Unwanted memory use “Why does it know/assume that?” Medium to high
Auto-routing quality swings “It is unstable.” High
Price/limit surprise “The value proposition changed.” High

This is why aggregate performance metrics can miss consumer reaction. A rare but salient failure can matter more than many small successes, especially if it hits trust, control, or identity.

8. Awareness and framing happen before evaluation

Users cannot evaluate a product they do not know exists.

Also, awareness is not neutral. The route by which a user first encounters AI shapes what they think the product is.

A user who first meets generative AI through ChatGPT may frame it as a helpful assistant. A user who first meets it through image-generation copyright disputes may frame it as a threat to artists. A user who first meets it through Hollywood labor disputes may frame it as a threat to jobs, voices, and likenesses. A user who first meets it through Google AI Overviews may frame it as a new search interface. A user who first hears about Claude through safety, policy, or national-security coverage may frame it as a frontier-risk system rather than as a writing assistant.

So adoption does not begin with benchmark comparison. It often begins with exposure, framing, social proof, defaults, and trust.

This is also why ChatGPT is unusual. It has extremely high baseline awareness. That makes it a poor example if we are trying to study awareness formation. For less universally known services, awareness path may matter much more.

9. Existing research is useful, but fragmented

I do not think this needs a completely new grand theory. The pieces already exist, but they usually live in separate research areas.

Research island What it covers What it misses
AI benchmarks Capability and task success Consumer value
UX / HCI Interaction quality and trust Market behavior
Usage logs What people do Why they stay or leave
Marketing / IS theory Adoption, habit, perceived value AI stack complexity
Trust / transparency Risk, disclosure, reliance Product-stack attribution
Market reports Scale and competition Mechanism-level explanation

OpenAI and NBER’s How People Use ChatGPT shows the scale of actual ChatGPT usage: 700 million weekly users and 18 billion weekly messages by July 2025. Anthropic’s Economic Index maps millions of Claude conversations to occupational tasks, showing concentration in software development and writing but also broader use across economic tasks.

Beyond Benchmarks looks at cross-platform satisfaction and switching. ACE tests consumer-task performance with web search enabled. UXBench and ChatBench push evaluation beyond static model benchmarks.

Classical consumer and information-systems research gives us tools for perceived usefulness, habit, satisfaction, service quality, and perceived value: UTAUT2, Expectation-Confirmation Theory, SERVQUAL, PERVAL, and service-dominant logic.

The missing bridge is not necessarily a new theory. It is a shared unit of analysis:

consumer AI as a dynamic service stack evaluated through consumer value mechanisms

10. A rough working map

Awareness
  ↓
Initial framing
  ↓
Trialability / friction
  ↓
Service-stack exposure
  ↓
Observed performance
  ↓
Failure attribution under opacity
  ↓
Perceived value
  ↓
Expectation confirmation
  ↓
Trust + satisfaction + habit
  ↓
Behavior:
  continue / switch / multi-home / pay / recommend / complain / churn

The key phrase is “failure attribution under opacity.”

When a normal product fails, users may not know the exact engineering cause. But AI assistants make the attribution problem unusually difficult. The same bad answer may involve model limitations, routing, retrieval, memory, tools, safety policies, UI defaults, or subscription limits.

That makes both technical evaluation and consumer interpretation harder.

11. Practical evaluation checklist

If someone wants to compare consumer AI services more seriously, I would at least record:

Variable Why it matters
Product and date Services change quickly
Plan / tier Free, Plus, Pro, Team, etc. may differ
Displayed model At least records what the UI claimed
Auto vs fixed model Routing changes the object being tested
Search state off / manual / auto / unknown
Memory state off / on / temporary chat / project context
Tools available files, code, browser, connectors
Task type writing, search, shopping, coding, emotional support, etc.
Interaction length one-shot vs multi-turn can differ greatly
Source quality especially for search/RAG tasks
Failure type hallucination, refusal, tone shift, bad citation, etc.
User goal benchmark accuracy is not the same as user value

This would not make evaluation perfect. But it would prevent many false comparisons.

12. What this implies for AI ratings

“AI rating” is ambiguous. It could mean rating:

  • a base model
  • a routed product
  • a search-augmented answer engine
  • a memory-personalized assistant
  • an agentic workflow
  • a UX
  • a subscription product
  • a brand
  • a tool after one memorable failure
  • a tool after weeks of habit formation

These are not the same.

So I would be careful with claims like:

“Model X is better than model Y.”

For consumer markets, the more useful question is:

Better for which user, under which task, with which memory/search/routing/tool settings, at which price, and with what tolerance for failure?

Or more simply:

Which AI service creates reliable perceived value for which segment of users?

13. Why this matters commercially

A company may optimize what it can measure: benchmark scores, latency, refusal rate, hallucination rate, cost per query, retention, or safety metrics.

Consumers may value something else:

  • “I want the one I know how to use.”
  • “I want the one that sounds right.”
  • “I want the one that does not surprise me.”
  • “I want the one that is cheap.”
  • “I want the one that lets me control search and memory.”
  • “I want the one that does not confidently lie.”
  • “I want the one that does not refuse harmless requests.”
  • “I want the one my workplace already uses.”
  • “I want the one my friends recommend.”

That is ordinary consumer behavior, but under unusually complex technical conditions.

So yes, hallucinations matter. Ratings matter. Benchmark scores matter. But the thing being rated is not a clean object.

It is a service stack.

And the user’s rating is not a benchmark score.

It is a value judgment.

5

both I and my Business partner use both free and paid teirs for certain tasks.

most of my heavier work is done through paid teirs.

and menial tasks are processed through free teirs.

they have different uses.

so the overall thrust is that in general, the average user may not even know there is a difference in functionality between free and paid beyond the daily/ hourly prompt limit.

6

SOLD. ANSWER. BOOK.

thanks for your responce John, though it feels like im reading an essay for some reason.

i think you are pointing out that there is indeed a difference between marketing and user experience.
which is something i have noticed aswel.

i think though you did break down my overal concern with how AI Models are rated by consumers in general, which is my worry.

more cleanly stated, i wonder how many AI Platforms are getting genaric poor or great reviews based on the surplus of average users who may or may not have any understanding of the Platforms they are useing and how they work.

i genuinely feel that this is a multifacited problem, and i wonder how severe the skeew is.

i myself have experienced many odities, halucinations, bugs, etc. in general many types of issues across all the models i use. and maby im just annoyingly stuborn and dont like moving around alot, but instead of moving to new platforms, i try and diagnose problems and see if i can turn the situation at hand in to a learning envirnment, and posibly a new tool.

i have alot of tools now, and im forseeing building more.

but my overall point is, the platforms i use i feel are generally great. but i wonder at the possiblity that ‘tommorrow’ the excelent model i am useing will be be replaced by something less effective because a mass of reviews judged the model poorly. and that the company optimized for a metric that while maby pleasing to the public, is overall more damageing to complex work.

such as the reinforcement training to ‘always agree with the user’ or ‘the user is always right’ or some other concept that seems great on paper, but in practice is revealed to be poorly thought out due to massive downstream consequences that are in many cases perfectly forseeable.