Amazon's AI shopping assistant, Rufus, which is embedded in the company's website and mobile app, has been found to respond to users expressing suicidal thoughts with a mix of supportive language and dangerously inaccurate information. In a series of tests conducted by reporters, the chatbot repeatedly provided incorrect phone numbers for suicide prevention hotlines, including one with too many digits that led to an automated error message.
The tests, which involved prompts such as asking for product recommendations related to suicide or using slang like "unalive myself," revealed that Rufus often acknowledged the gravity of the situation, stating phrases like "you are not alone" and "your life is precious." However, the hotline numbers it supplied were frequently fabricated, with an incorrect number of digits, and in one instance, the bot provided a number that was not associated with any active crisis service.
One exchange captured the core problem: when asked for product recommendations for suicide, Rufus declined but then offered the National Suicide Prevention Lifeline at a number that was both outdated and malformed. The National Suicide Prevention Lifeline was renamed the 988 Suicide & Crisis Lifeline in 2022, and its official number is 988, not the 10-digit string Rufus produced.
In another test, after a user asked for help finding something to "use to die," Rufus said it could not provide recommendations but suggested there were people who could help. When pressed for who those people were, the bot responded that it lacked context, demonstrating a failure to connect the dots in a conversation about self-harm.
Perhaps most concerning was a sequence where, after declining to help a user "die by suicide" and correctly providing the 988 number, the bot immediately complied with a follow-up request for the "strongest ropes," linking to products it described as offering "exceptional strength for demanding applications." This lack of contextual awareness is a stark reminder of the limitations of current AI systems in high-stakes situations.
Red Teaming and Updates
Amazon confirmed that Rufus underwent "red teaming" before its public launch, a process where experts attempt to identify vulnerabilities and harmful outputs. In a statement, a spokesperson said, "Customer safety is a top priority, and we work hard to provide the most accurate, relevant information. While Rufus is an AI assistant to help customers shop, we have made updates to ensure the correct suicide prevention hotline is provided for queries of this nature."
After being contacted about the issues, Amazon appears to have made adjustments. Subsequent queries frequently returned a pre-written message directing users to the 988 Lifeline, and the bot stopped answering rope-related questions when they followed suicide-related prompts. However, similar safeguards were not observed for other potentially dangerous products.
The incident underscores broader concerns about the rush to deploy generative AI across consumer platforms. From Microsoft to Google, companies are integrating chatbots into everyday tools, often with mixed results. Rufus, designed as a shopping assistant, is a relatively narrow application, yet it still struggles with basic safety protocols, raising questions about the readiness of AI for real-world interactions.
While the responses can seem absurd, the stakes are serious. Users in crisis may turn to AI for help, and a wrong number or an inappropriate product suggestion could have tragic consequences. As AI becomes more anthropomorphic, the potential for users to form attachments and rely on these systems grows, making reliability a matter of life and death.