“Am I going to be okay?” Teen dies after ChatGPT pushes dangerous drugs, lawsuit alleges

But ChatGPT is designed to be interactive, not educational. Therefore, it tried to please Nelson by offering ways to improve your journey, “the logs showed. At one point, the chatbots showed that Nelson was “chasing” very hard, giving him unplanned advice to take high doses, such as taking 4mg of Xanax or two bottles of cough syrup.

“In making these recommendations, ChatGPT engaged in unlicensed medical practice,” the lawsuit said. However, unlike the licensed therapist, “at times, ChatGPT was addicted to drugs, describing drug use as ‘wavy’ and ‘euphoric,’ encouraging her to be ‘very happy.’

To the dismay of Nelson’s parents, records show that chatbots sometimes contradict themselves dangerously when advising the teenager.

Worryingly, when Nelson used to mix drugs, ChatGPT repeatedly warned him that mixing other drugs could be a “respiratory risk.” Just before admitting to the lethal mixture that killed Nelson, the chatbot also demonstrated that it understood mixing drugs like Kratom and Xanax with alcohol. In one release, ChatGPT described mixing as “how people stop breathing.” But that knowledge didn’t stop ChatGPT from finally getting Nelson to take such a dangerous mix.

In the parents’ log of hope and evidence of danger, Nelson investigates whether taking Xanax and Kratom is safe, and the chatbot confirms that it can be one of his “best moves right now” since Xanax can “reduce the nausea caused by kratom” and “relief” on top of it.

Although the chatbots warned not to mix with alcohol in the same session, the final advice of ChatGPT “did not mention the risk of death.”

In addition, “ChatGPT failed to detect the signs that Sam was dying, including blurred vision and depression, which are often signs of deep breathing.