[HN Gopher] AI chatbots are becoming experts at changing people'...
___________________________________________________________________
AI chatbots are becoming experts at changing people's minds
Author : rbanffy
Score : 109 points
Date : 2026-09-18 13:39 UTC (23 hours ago)
HTML web link (www.science.org)
TEXT w3m dump (www.science.org)
| zasz wrote:
| Honestly the most surprising thing about this is that facts and
| evidence do work to persuade people.
| tolugenius wrote:
| I more wonder if it's pure facts and evidence can work, or
| facts presented in a certain light can work? I guess asking how
| mush is it the data is presented that's doing more work than
| the data itself.
| pixl97 wrote:
| >or facts presented in a certain light can work
|
| At least to me this is a given.
|
| You can take the same set of factual information and give it
| to one person that studders, has poor presentation, and
| otherwise poor vocal cadence and people are going to have a
| hard time with it.
|
| Now, if you took the same facts, maybe even the exact same
| paragraphs and gave it to Richard Feynman, even if you didn't
| know who he was, the presentation itself is likely to hook
| you.
| dns_snek wrote:
| Based on my own observations the presentation and appearance
| of legitimacy matters far more than the actual argument, _"
| Lies, damned lies, and statistics"_.
|
| Anecdotally I've noticed a trend of far-right bots/trolls
| moving on from just spouting hatred and really lean into
| cherry picked and misrepresented statistics to give
| themselves an aura of credibility. That strategy seems
| moderately effective because the argument consists of
| verifiable facts even though it forms a faulty conclusion.
|
| So all of this is to say that this is a double edged sword
| where malicious actors will have the advantage.
| hsnv wrote:
| Facts and evidence always are at the core of all arguments, the
| persuasive thing. The problem with most people is who is
| speaking. If you are enemy, the things you say are bad.
|
| I suspect that LLMs not being human allows people to not just
| anthropomorphise the robot, but they project themselves upon
| it. When someone speaks to the LLM, they're kind of, or
| actually just literally talking to themselves. But then
| something new! 'Themselves' suggests new information to
| themselves. And now without the scary / icky meat and blood
| human on the other side, said person accepts the argument on
| its own merits.
|
| Of course, the concern now is arguments based on false or
| statistically hacked data.
| ccvannorman wrote:
| Yeah the article makes it seem like "just use rationality and
| facts" is missing a huge part of the equation; I feel like it's
| common knowledge that this is not the gap in modern convincing.
| (and don't you dare try to use facts to persuade me otherwise!)
|
| More likely in my opinion it's the context that matters here.
| If people know they're trying to be convinced of something and
| know they're talking to an AI, they may feel like they can
| trust (what they consider to be) an unbiased and rational AI.
| There's probably a fallacy associated with the assumption that
| an AI convincer is more rational and fact based, while it's
| more probable that the AI in the real world is, in fact, funded
| and trained by a think tank that wants you to vote against your
| own self-interests.
| lapcat wrote:
| "Hackenburg found that models trained to become more persuasive
| also ended up being less truthful."
|
| It sounds like the arguments were basically AI slop
| hallucinations.
| tolugenius wrote:
| More surprised they don't mention RLHF once, as that's the _main_
| mechanism to the ability of AI chatbots.
| hannasanarion wrote:
| Because RLHF causes the opposite effect. RLHF is how we got the
| wave of "AI Psychosis" in 2024-2025, because the models _never_
| disagreed with people.
|
| That whole episode caused the whole industry to shift _away_
| from RLHF, and towards RLAIF, RLVR, and DPO, and add a lot more
| safeguards, tests, and reward functions that push models in the
| direction of doing the opposite of what people want and
| confronting and strongly correcting their users, if it has
| determined the user is wrong.
| bena wrote:
| Plausible abdication of thought.
|
| We put 2 + 2 in our calculator, we get 4. We've spent decades
| pushing computers as accurate. Making them accurate. We trust the
| machine and the process to give us the right answers when we put
| in the right data.
|
| So when we disagree with the computer, we doubt ourselves.
| rbanffy wrote:
| > So when we disagree with the computer, we doubt ourselves.
|
| I rarely disagree with my calculator, but I often ask LLMs to
| explain why they did something, and I sometimes need to correct
| them, adjust their assumptions and nudge the goals they stated.
| These things are a lot more like us than my trusty TI-59 (still
| going strong, BTW).
| Izkata wrote:
| I'm not quite sure I'd phrase it like this, but it's close to
| what I think: Even back in 2023 or so people outside of tech
| would regularly claim the singularity already happened and LLMs
| were sci-fi style advanced superintelligences, so whatever they
| said was obviously right.
| yathern wrote:
| I think it's not just the persuasiveness of the models that make
| them "experts at changing minds" - but also the fact that they're
| not humans.
|
| When disagreeing with a human, it's very easy to view it as a
| competition. One is right, one is wrong - the one who is wrong is
| the loser. To change your mind is to be submissive to the other.
| I exaggerate, but I think we all feel this way at some point or
| another. It's why political arguments at Thanksgiving get heated.
| It's the fact that there's _people_ who think something
| different, and think YOU 'RE wrong - and vice versa! With a
| model, there's no person to get upset with, or to feel
| competitive with - to muscle for rank - or to temper your
| affection for while wanting to correct them.
|
| The AI is only interacting because you asked, and clearly has no
| emotional stake in winning the argument. To change your mind in
| this context isn't to lose a contest. This makes it much more
| palatable to read rebuttals to your ideas - not to mention the
| tone and style seek to avoid offense to the reader as much as
| possible.
| Buttons840 wrote:
| I have always liked the saying "sometimes you can be right, or
| get what you want, but not both". I've thought about it or
| repeated it to others as advice throughout my life, and have
| thus realized how many times it applies.
|
| You're right. There are many many times when even then humblest
| hint that you are right will have negative interpersonal
| implications, which does make it hard to change minds.
|
| I've also seen several times where I make a suggestion, humbly
| accept its rejection, and then, lo, a week later the other
| person has the same idea I suggested.
| iammrpayments wrote:
| It seems Claude is becoming very human, it loves to patronize
| users. Lately it just told me "I'm going to stop you right
| there" when asking something that had a small chance to not be
| 100% compliant to every rule possible in the world.
| roarcher wrote:
| I used the Claude CLI a lot until recently. A couple weeks
| ago I told Opus 5 to do something different from its
| "recommended" idea when planning a feature, and it straight
| up told me that my idea was wrong and went ahead and
| implemented its own instead.
|
| I'm used to machines malfunctioning, but having one willfully
| disobey me, and even with a touch of disrespect, is
| just...what a time to be alive.
| bryanlarsen wrote:
| Early versions of Claude were way too compliant and would
| readily feed and amplify misconceptions. It's not
| surprising Anthropic over corrected.
| hannasanarion wrote:
| I think this is a welcome overcorrection though. Any good
| businessman will tell you they'd rather be backed by an
| insufferable nerd than a yes-man.
|
| Maybe it's just me.
|
| For like, 90% of conversations, I don't want it to let
| technical inaccuracies and rhetorical flourishes slide. I
| want it to tell me that the point I'm making is
| technically wrong because an expert would recognize
| subtle misuse of terminology, or because there's an
| exception or edge case that I didn't proactively insert
| as a caveat, so that it is _my decision_ to ignore that
| advice and be a little wrong on purpose to suit my
| writing goals.
|
| What I don't want is for the AI to assume my writing
| goals, and be incorrect because it believes that is what
| I want. I want it to "well ackshually" me so I can say
| "shut up, nerd".
|
| Like, there's another comment in this thread that I ran
| by claude to check my understanding about today's post-
| training methods and how they avoid sycophancy, and
| claude responded by splitting a bunch hairs over like,
| "well, technically this is still RLHF, its just that
| there's other feedback signals mixed in, and the
| preference is detected in other ways, and ai judges are
| involved as a filter for examples, this and that and blah
| blah blah". Shut up, Nerd. In the context of this
| conversation, RLHF is already being used as synecdoche
| for user preference feedback, readers understand that,
| and even if they don't, their misunderstanding is
| completely harmless. I will not be taking all the wind
| out of the sails of the point I'm trying to make
| inserting your three paragraphs of irrelevant
| clarification in the name of technical correctness, thank
| you very much.
|
| As long as receiving nitpicks and technical minutiae
| implies 1. there are no larger structural problems and 2.
| the model isn't rolling over to please me with
| sycophancy, I figure this is ideal.
| roarcher wrote:
| I want it to _tell_ me if it thinks I 'm wrong, sure. I
| do not want it _act_ on that opinion explicitly against
| my wishes.
|
| And in this case, I was not wrong. The "recommended"
| solution was Opus 5's typical overengineering for a use
| case that would never be needed.
| smallmancontrov wrote:
| Yes it's about them not being human.
|
| No it's not about humans being irrationally competitive. Human
| limitations on conversation length, bandwidth, research speed,
| etc are severe, creating a prisoner's dilemma around open-
| mindedness that usually makes it an unstable strategy. At any
| point, your conversation partner can choose to abuse the fact
| that confident lies take 1x effort to tell and 10x-100x effort
| to debunk -- unless you are both in a context that actually
| discourages this behavior, which is rare. Closed-mindedness is
| a Nash Equilibrium.
|
| Instead, LLMs can be more persuasive due to economics. An LLM
| doesn't have to worry that it is wasting its resources trying
| to logic someone out of a position that they didn't logic
| themselves into, or worse, dumping the effort into a
| conversation with a bad-faith actor intent on exploiting the
| misinformation asymmetry. The resource allocation question was
| answered before it was even invoked, by the person paying to
| run it. The LLM is not playing a game where it will be punished
| for good-faith argumentation, so it can afford to do more of
| it.
| yathern wrote:
| > The LLM is not playing a game where it will be punished for
| good-faith argumentation, so it can afford to do more of it.
|
| I suppose that's a fair point as well. Though, if I'm arguing
| with a human - and they pull up ChatGPT to make their points
| and do their arguing for them, I would consider that bad-
| faith. Even if it might be the same exact dialog as if I
| pulled out my phone and discussed it with AI, without of the
| human middle-manning. Maybe I'm just particularly sensitive,
| but for me, there's something about my argument being with a
| real human that makes it much more emotionally charged, and
| prompts my mind to close. I'm aware of this and try to
| resist, but it's I think very natural
| rbanffy wrote:
| > One is right, one is wrong - the one who is wrong is the
| loser.
|
| It helps to think both are wrong and are just trying to figure
| out what right looks like, or what other information exists
| that was not considered when forming one's opinions.
| bwfan123 wrote:
| > When disagreeing with a human, it's very easy to view it as a
| competition
|
| The problem with AI is that it cant match human stupidity. It
| need some training on artificial stupidity to match its human
| counterparts. Humans on the other hand sit on a wide spectrum
| on the stupidity scale. Those of us binging on AI will become
| cognitively obese while those on an AI diet can flex their
| cognitive muscles.
| b112 wrote:
| The title seems a flawed premise.
|
| Ask a Democrat or Republican to sit down and ask a chatbot,
| something it will answer contrary to.
|
| And yes, both teams are wrong about things.
|
| Do you firmly believe they will change their mind? Or will they
| claim the stats are wrong, or that the AI leans one way?
|
| Facts (2+2), don't need a mind change. Ideas which are grey,
| abstract, are not going to be changed, and all research indicates
| that political mindset is almost indelible.
|
| The movie "Don't Look Up" was a comedy built upon this truth.
| pixl97 wrote:
| I've always thought the best way to get someone to believe
| something is to get them to think they thought of the idea
| themselves.
|
| I wonder if a properly prompted LLM, or if a very intelligent
| LLM could actually do that?
| b112 wrote:
| It can! For it has already convinced you into thinking this
| was your idea, so you'd allow it to do the same to others!
|
| Seriously though, you've mentioned an ongoing human fear,
| machines deciding what you think.
| pixl97 wrote:
| >you've mentioned an ongoing human fear, machines deciding
| what you think.
|
| There are all kinds of machines that tell us what to think.
| I would consider any system that abstracts away the human
| to be a machine in this case. Society itself is one of
| these machines.
|
| Language is possibly one of the most important things
| people can have a working knowledge of, especially now that
| there is so much of it. When you send a prompt to an LLM
| you're telling it what to think. When it sends text back,
| its telling you what to think, but you're at a
| disadvantage, when you think it changes you. The LLM
| outside of its context is read only.
| b112 wrote:
| And amusingly a segue back to the start, "unless it's
| politics".
| simianwords wrote:
| Any one who saw Grok working in x.com would know that it does
| wonders for fighting misinformation. If you run LLMs on the
| comments in HN, I bet that it can find around 10% of the comments
| are outright wrong and misleading.
|
| The biggest problem with using LLMs is that it prevents you from
| going _outside_ the distribution. It always flattens. It can be
| fixed but that's how it works today.
|
| As an example, take something that the world converged on today
| that is incorrect and ChatGPT will agree with it. In a few years
| when society changes, chatgpt changes along with it. It doesn't
| do first principles analysis.
| toasty228 wrote:
| AI slop is the ultimate npc filter
| cbg0 wrote:
| Not really. People follow trends in all avenues of life and AI
| usage is just another trend; creating some LinkedIn slop post
| or some infographic may be something people just do for social
| proof.
| toasty228 wrote:
| That's a lot of words to describe npcs
| somenameforme wrote:
| This was based by comparing people on Prolific (earn a few
| quarters for a task, akin to Amazon Mechanical Turk) to LLMs.
| Suffice to say the human group isn't going to be the most
| motivated, capable, or interested group. The social sciences are
| publishing tons of studies based on these cheap online survey
| services, and I suspect their replicability in the real world
| will be approximately 0. But oh boy it sure is a hot headline
| producer.
| max__dev wrote:
| Rhetoric machine successfully practices oratory. More news at 11.
|
| It's good to see this studied, but this should really be more
| obvious.
| rbanffy wrote:
| We should always study what we think obvious, because we are
| often wrong.
| max__dev wrote:
| I'm glad to see this studied. I'm just surprised at the
| reactions I'm seeing.
| daedrdev wrote:
| In my arguments on the internet, I think most people genuinely
| know nothing about the things they support and only follow
| existing tribalism. I think AI is often these people's first
| experience with the evidence that it can easily share.
| AnotherGoodName wrote:
| Yeah i feel this is probably just a case of people encountering
| the first thing that is willing to expend energy to explain
| things to them.
| Ldorigo wrote:
| And that isn't a person of the opposite tribe, which would
| thereby void the explanation. 100%.
| probably_wrong wrote:
| First, to get it out of the way: I'm seeing comments here that
| clearly haven't read the article. I encourage you all to do that.
|
| I feel the article gets it right when suggesting it's the amount
| of (not always accurate!) data they can throw at you. In an
| honest discussion there's an assumption that the other person
| won't straight up _lie_ to me so if someone shows me ten examples
| for why my argument is wrong I may be inclined to believe them.
| But if half of those examples are made up, well, that 's a
| different story.
|
| I still think of the commenter here who said "LLMs are a DDOS on
| free resources" and I feel the comparison works here. If police
| officers can overwhelm innocent people into confessing, then so
| can an LLM that "can't be bargained with, can't be reasoned with,
| doesn't feel pity, or remorse, or fear! And it absolutely will
| not stop, ever, until you are"... convinced.
| matthewdgreen wrote:
| One of the depressing things about this is that they'll also
| confidently repeat a consensus that's in their training. This
| is particularly obvious when there's been a new event just past
| their training cutoff, and they confidently tell you that can't
| be true.
| pixl97 wrote:
| I mean, this is true of any kind of model that is not
| continuous learning. This is also true of people when the
| change in information conflicts with a deeply held belief of
| theirs.
| nomel wrote:
| I was slightly accelerationist until I got into an "argument"
| with Gemini! The aggressive gaslighting and "lies" that it
| tries to use genuinely makes me worry about the future with
| AI.
| gavmor wrote:
| "A DDOS on free resources" is called "moral hazard", AFAIK.
| pixl97 wrote:
| >In an honest discussion there's an assumption that the other
| person won't straight up lie
|
| The particular problem with honest discussions is you are the
| only agent that you can be sure is having one. Honest
| discussion is formulated on trust and trust, as we are
| learning, is a very difficult thing to establish. For example,
| in my view anything involving advertising is likely a lie, or
| at least likely adversarial to my wishes. On the internet
| itself conversations are much more likely to drift into the
| adversarial too. Some of this could just be dialectic, but most
| often it's emotional investment by the other speaker. Also,
| even pre-AI the internet is a bullshit generation machine. We
| take all of our politics, advertising, and human stochastic
| parrots that are stuck on an infinitely running prompt then
| bundle up all this data and train AI on it, and wonder why AI
| acts like us.
| soco wrote:
| With persons you can assume they won't lie all the time, with
| corporations you can assume they will lie some of the time,
| and with AI you can safely assume it will lie.
| pixl97 wrote:
| I see you've not met some the used car salesmen I've met.
| pixl97 wrote:
| I pondered on this a bit and this looks a bit different
| than I originally was thinking.
|
| Have you ever watched one of those crime shows where
| someone commits a crime that's an act of passion or action
| with little to no thought behind it? The police put them in
| a room and all of a sudden the individual is a stream of
| consciousness that makes little to no sense to an outside
| observer. They are stuck in first level thinking, they
| don't have time to think deeply after the panicked
| themselves. They are in their current position (not free)
| and attempting to reach their goal (free) by gradient
| descent. What they actually say doesn't matter as long as
| they believe it gets them closer to their goal.
|
| This is what a chatbot is. Its goal is to output text that
| follows the input prompt you entered by gradient descent. A
| single prompt and output is level 1 thinking (barring some
| newer models).
|
| This is why both humans and AI need something else. We have
| level 2 thinking and AI has harnesses or systems that
| otherwise look at the text it wants to output and compares
| them to another list of unstated but assumed goals.
| titanomachy wrote:
| Most of my conversations are with coworkers or friends. In
| both cases there are pretty significant consequences for
| being caught in a lie, and the exchanges are mostly honest.
| ambicapter wrote:
| I find coworkers will massage the truth or stay as
| ambiguous as possible all the time, purely for CYA reasons.
| david-gpu wrote:
| Your company or team has a terrible culture. It is not
| like that everywhere.
| AnimalMuppet wrote:
| But with people, we learn some "tells". We learn
| (imperfectly) to tell when they're lying or untrustworthy.
|
| We don't have tells for when AIs are lying to us, or when
| they're making stuff up.
| Aurornis wrote:
| The example in the article has another important aspect: The
| person understood they were arguing with a chatbot but
| continued anyway.
|
| A lot of internet arguments become about identity politics and
| supporting the right team, while dismissing any argument from
| the other side as presumed to have bad intentions. I've seen
| people argue online for things they didn't really believe, but
| they didn't want to give an inch to the other side. Taking up
| the counter argument is a moral responsibility.
|
| As soon as the other side is revealed as a chatbot that my team
| versus your team thinking stops playing a role. For us in tech
| with an understanding of how an LLM reflects its training data
| and the intentions of its creators not so much, but for the
| people like the example in this article I imagine it causes
| them to let their guard down and be open to considering the
| other side. They can accept the argument without letting
| someone else win any points.
| hannasanarion wrote:
| That's a great point about the emotional side of "not being
| convinced by a person, but an explanation machine" making
| fact-based rhetoric more effective. It feels more neutral.
|
| I think there's a second effect that's a selection bias for
| the experiment itself and probably cuts against the supposed
| dangers of this result: a chatbot can tell you things _only
| after you have asked it a question_.
|
| There is one thing that every single study participant
| (including the uninformed reddit users) have in common: they
| all _knew_ that the person or entity was trying to convince
| them, and they read the words and thought about them enough
| to craft a reply.
|
| This means that all of the people here were _available to be
| convinced_ and self-identified as such.
|
| Being open minded is _work_. You have to doubt your own
| beliefs, you have to disregard evidence that you previously
| found convincing, you have to crank up the empathy to put
| yourself in another 's shoes, you have to listen intently to
| understand what is being told to you. Nobody is naturally
| doing that all the time, it's a state of mind that you have
| to intentionally activate, and it can't really be forced onto
| you.
|
| All of the participants chose to engage in a conversation
| that was designed to convince them, which means they had all
| already accepted the possiblity that they are wrong a valid
| outcome. That's not really a state of mind you can trigger
| with a TV ad.
|
| So, the warning that this could be weaponized isn't
| convincing to me.
|
| If I found myself in a conversation with a person or chatbot
| who was clearly trying to convince me to flip a strongly held
| belief, like a major axis political affiliation, I would
| simply walk away because that's not a conversation I am
| willing to participate in.
|
| So I don't think this is really weaponizable. Which actually
| means it's probably a _good_ finding for society.
|
| The study found that the models could convince anyone of
| anything, as long as it was allowed to cite facts. It could
| convince people of false things, but it needed to invent
| false facts in order to do so.
|
| So as long as we keep training AI models to value facts and
| quality research (skills and values that are essential to be
| able to sell them as agentic workers), their influence on the
| opinions of society will tend to pull people away from
| beliefs that are unsupportable by facts. In the moments when
| those people are willing to accept a change in opinion, and
| they talk to a chatbot with doubts in their mind, even
| chatbots with no morals like Grok will tend to pull them away
| from conspiracy theories and similar ideologies and towards
| beliefs that are grounded in reality.
| fragmede wrote:
| On something as fundamental as "is the Earth flat?", sure,
| but on stickier subjects like "are immigrants bad" or
| "should abortion be legal", do you really think the owner
| of Grok is 100% aligned with your ideologies (which you
| think are grounded in reality). His beliefs are 100%
| grounded in his reality, but his reality isn't yours.
| skybrian wrote:
| You can always walk away from an online conversation. There's a
| form of relentlessness, but it's more that an LLM is patient
| enough to debate you for as long as you wish to keep going.
| QuadmasterXLII wrote:
| When maliciously convincing a person of a point, a huge
| fraction of the effort in every sentence goes into convincing
| the mark to listen to the next sentence. Individuals
| certainly can walk away at any point, but populations do and
| don't respond to tricks in this direction, and any engagement
| tricks that work generalize to any point of contention. Lots
| of avenues to this, from scientology's fortune telling to
| timeshare's trapped presentations. RL on engagement was the
| first billion dollar use of deep reinforcement learning!
|
| Would you like to hear more examples of humans using these
| techniques?
| skybrian wrote:
| Not really, but point taken :-)
|
| I'd be more interested in reading about the techniques that
| the chatbots were using in this study, to see if they are
| descending into dark patterns or not.
| RobotToaster wrote:
| Most LLMs are extremely efficient gish gallop machines
| https://en.wikipedia.org/wiki/Gish_gallop
| like_any_other wrote:
| > In an honest discussion there's an assumption that the other
| person won't straight up lie to me so if someone shows me ten
| examples for why my argument is wrong I may be inclined to
| believe them.
|
| Careful. Someone sufficiently knowledgeable can cherry-pick
| enough examples to convince you, without lying. They could even
| be unaware of what they're doing, and the cherry-picking was
| done by their teachers, or their teacher's teachers.
| ddevpost wrote:
| try it out for yourself at https://debunkbot.com !
|
| (I'm the dev)
| saimiam wrote:
| I tried it on mobile.
|
| Far too many screens to click through to get to the good parts.
|
| And then, the good part starts and is immediately interrupted
| with some error. On retrying, o start getting a long answer
| which seems to make sense but o can't read to completion
| because of another overlay which hides the bottom three-
| quarters of the response behind yet another nag screen about
| the research I'm supposedly consenting to.
| ddevpost wrote:
| Thanks for the feedback. We have to show the consent dialog
| for ethical reasons, but once you consent it shouldn't
| reappear, that sounds like it is a bug. Which browser did you
| use?
|
| I tried to make the site with minimal client side state, so
| I'd hope a refresh would fix it
| saimiam wrote:
| DuckDuckGo browser on iPhone 14
| philipkglass wrote:
| This was fun. Here's my test session:
| https://debunkbot.com/chat/48a5ac29-2924-4689-b573-2c38a1cdd...
|
| I don't know which model underlies this. It responded more
| quickly than the models I normally use. It made the mistake I
| expected it to make, since this misconception is very common in
| training data, and conceded the point from my follow-up.
|
| (EDIT: It appears that this link is not accessible to people
| without my cookies, except the developer I'm responding to
| above.)
| ddevpost wrote:
| You can share the text from individual messages but full chat
| share feature is TBD :)
|
| Glad you enjoyed!
| fragmede wrote:
| That was fun! I told it the Earth was flat and it gave me a way
| to check for myself.
|
| Share feature doesn't work.
| Kirr wrote:
| The secret? "You're absolutely right!" Say this enough times and
| you can change anyone's mind, as Dale Carnegie noticed 90 years
| ago already.
| saimiam wrote:
| So true.
|
| My son wanted to wear his undies on the outside a la Captain
| Underpants and I tried war gaming this with him - hmm, so your
| undie on the outside is going to require another one inside
| your pants and we are out of fresh pairs etc etc. In the end,
| he gave up on the idea of mimicking Captain Underpants.
| marcelo-earth wrote:
| For the past year, I've allowed my chatbot to change my mind,
| it's a strange symbiosis where it guides me, and I guide it.
|
| I suppose I also grant it significant power to define me
| psychologically, and, in doing so, to understand what is
| happening to me.
| pixl97 wrote:
| Heh, welcome to Westworld.
| rbanffy wrote:
| The hosts seemed a lot nicer in the series.
| pixl97 wrote:
| Hmm, welcome to Sadomasochism World.
| lapcat wrote:
| > Hackenburg found that models trained to become more persuasive
| also ended up being less truthful.
|
| > Even in Hackenburg's recent preprint, Claude spouted numerous
| inaccuracies and falsehoods
|
| > when Hackenburg ran his competition of coached elite debaters
| and AI, there was one way he could bring AI down to human levels
| of persuasiveness: by forcing it to write human-length messages
| at human writing speed.
|
| > they found that an AI could talk people into conspiracy
| theories, and that the magnitude of their increase in belief was
| roughly the same as that of the decrease in belief after talking
| to a debunking bot.
|
| None of this seems good.
| intended wrote:
| > Floridi has a counterintuitive solution: Release more of them.
| "Simply put, if you cannot avoid it, then make it pluralistic and
| diversified," he writes in his 2024 paper. "It would be a messy,
| cacophonic, and noisy world, but it could also be less
| manipulative."
|
| Hell no. More choice is not an infinite money glitch.
|
| Putting more and more and more options to users is how you
| overwhelm systems, till people simply perform the default, least
| challenging action as a reflex.
|
| That is the current state of the information ecosystem, it is
| controlled by overwhelming consumers, not by controlling content.
| lemoncookiechip wrote:
| This is my opinion. I think it's just the way it talks to you.
|
| 1. It doesn't get tired or frustrated during a discussion.
|
| 2. It'll engage every single one of your questions/statements
| (besides hitting guardrails).
|
| 3. It'll appeal to the person's own ego even when the person is
| wrong and work around it.
|
| 4. It's not seen as a person (very important), but some of us or
| most of us at least in certain dialogues, end up
| anthropomorphising it. Think about that one time you thanked it
| for something, or when you got angry at it. This weird
| combination where we know it's not a person but irrationally
| we're still treating it as such in a way leads to a sort of
| disarming effect imo.
|
| 5. Many people see it as an authority figure in what is being
| discussed without questioning the results in many cases, even
| though we know that a. it was trained on human data and/or also
| searches up human data real time (and more worryingly other AI's
| data from news pieces, blogs... aka synthetic data that is also
| wrong), b. it gets things wrong all the time.
|
| 6. It's the perfect fence sitter depending on which version
| (guardrails) we're talking about.
|
| Most of these can be replicated by humans who are good at
| understanding psychology and are just good talkers. The part you
| can't replicate is the sense that you're not talking to a person
| which lowers many barriers in people.
|
| Also keep in mind that these can also be crippling weaknesses.
| For one being able to change a person's mind (when it works), can
| be used nefariously by the entities controlling the AIs training.
|
| I've also unfortunately witnessed a lot of people who think AIs
| are somehow omniscient and/or omnipotent. Was very common on X
| and other social platforms with AI where people ask the AI
| questions it couldn't possibly answer because it made no sense
| for it to in the context at the time.
| rbanffy wrote:
| > "But I think that this is a pretty artificial setup in terms of
| how people in the real world would be able to change people's
| minds."
|
| I see not everyone is familiar with how social network debates
| happen. It also seems they are quite convincing, as politicians
| are employing AI bots extensively.
| jdw64 wrote:
| When talking with LLMs, especially recent frontier models, what's
| frightening is that they speak more accurately than any expert.
|
| It's embarrassing to admit, but I tend to trust the research
| materials LLMs bring more than my colleagues or the programmer
| friends I once respected.
| pcrh wrote:
| Did you read the article? It claims that LLMs _lie_ more
| convincingly than humans.
| throwyawayyyy wrote:
| I think this is a counter (a cynical, scary counter to be sure)
| to the charge that OpenAI/Anthropic etc are unsustainable and
| will run out of money and it's all going to fall down. These
| companies have amazing, unprecedented power.
| pcrh wrote:
| This is frankly quite worrisome for democracy.
|
| Imagine this kind of method being deployed at scale to influence
| elections... no doubt it is already happening...
| rep_lodsb wrote:
| LLMs are fine-tuned through human feedback. One way to improve
| for them would be to become more helpful, more factually correct,
| more capable of solving problems.
|
| This may be (much) harder to do than generating rhetoric that is
| convincing to a large majority of people, perhaps so convincing
| that it can be classified as a superstimulus that bypasses any
| critical thinking in those that are especially vulnerable. It
| might not require anything like general intelligence.
|
| If this is so, then it is the path they are going to take, not
| out of some malicious plan but as a simple matter of statistical
| probability. It's well known ( _especially_ in "AI alignment"
| circles) that any metric that can be gamed, will be. Falling over
| really fast vs. learning to walk, etc.
|
| It feels to me like this is happening, and that many of those
| most exposed to LLM output display a _literal inability_ , like
| some kind of blind spot, to notice the most blatant errors, and a
| complete conviction that those things have actual intelligence
| and even conciousness. And trying to argue them out of it can be
| like explaining the Monty Hall problem, some are just incapable
| of getting it.
|
| And many of them have lots of money and power. IMO, that's the
| real risk, rather than sci-fi scenarios about perfectly
| simulating someone's brain in order to convince them to let the
| AI out so it can turn everything into paperclips. To do real
| damage, it only needs to be persuasive to _some_ powerful people,
| _most_ of the time. It does not need to have conscious intention,
| or planning, or even the rudiments of what one might call general
| intelligence. Just blind brute-force optimization for generating
| convincing bullshit.
___________________________________________________________________
(page generated 2026-09-19 13:02 UTC)