AI just can't be trusted with anything outside of maybe creative stuff and some simple coding. It just very easily creates things that are false, but talk about them with such confidence that makes some people actually believe it.
This is the crux of it tbh. LLMs are sophists, and worse, they're sycophants; if you're using an LLM for advice/feedback/psychotherapy/companionship—i.e. instances where there
are no clear-cut objective answers to prompts, you're inevitably going to get responses that are a) very convincing and b) very biased in
your favour. If LLMs will validate and even create psychotic delusions (and they have, an alarming amount of times)
because the user is always good/smart/right it's probably not going to tell you the truth at the expense of your feelings.
I used to think you could get around this behaviour by simply adding instructions to be brutally honest or combative in the LLM's custom prompt, but because its primary goal is still making you feel good, all this means is that it'll push back slightly or vaguely before returning to default mode (and then telling you how wonderful and smart you are for wanting the truth at the expense of your feelings, lol).
Hardly any original thoughts. But I used to think these concerns were overblown, or at least lacked nuance. You know things are bad when
AI-induced psychosis is an actual term. I do wonder if the user-glazing behaviour could be avoided by hosting one locally.
not like they'd believe it was the crippled guy who did it.
