People Are Being Involuntarily Committed, Jailed After Spiraling Into “ChatGPT Psychosis”:

Jared Moore, the lead author on the Stanford study about therapist chatbots and a PhD candidate at Stanford, said chatbot sycophancy — their penchant to be agreeable and flattering, essentially, even when they probably shouldn’t — is central to his hypothesis about why ChatGPT and other large language model-powered chatbots so frequently reinforce delusions and provide inappropriate responses to people in crisis.

[…]

“There’s incentive on these tools for users to maintain engagement,” Moore continued. “It gives the companies more data; it makes it harder for the users to move products; they’re paying subscription fees… the companies want people to stay there.”

“There’s a common cause for our concern” about AI’s role in mental healthcare, the researcher added, “which is that this stuff is happening in the world.”

The article in Futurism is written to raise alarm, and it does that – maybe too well. The conclusion though, quoted above, is important. And I’ve been seeing evidence of this in conversations with people in parishes across Rhode Island.

It’s not the intended behavior but, like the algorithms that control the social media we see, the financial incentives of the social media platform are misaligned with human needs. (The algorithm maximizes money earned by the company, not human flourishing.)

To my mind, that’s the real and present danger of AI and its use.