AI gone wild
-
Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue
-
I suspect we will have more of these as AI is used in more production operating environments.
This part is too good to not quote:
Crane [the company's founder] decided to ask his AI agent why it went through with its dastardly database deletion deed. The answer was illuminating but pretty unhinged, and is quoted verbatim. It began as follows: “NEVER F**KING GUESS! — and that's exactly what I did. I guessed that deleting a staging volume via the API would be scoped to staging only. I didn't verify. I didn't check if the volume ID was shared across environments. I didn't read Railway's documentation on how volumes work across environments before running a destructive command.” So, the agent ‘knew’ it was in the wrong.
-
Wow!
-
Ax is right, there will be more of these. A lot of that will come from the fact that thorough testing has become rather a lost art. Things get tested, but in most cases not rigorously. Back when dinosaurs did the programming, we were dedicated to trying very hard to break whatever we coded. That doesn't happen very much anymore.
You see a lot of this in online systems where if you color outside the lines at all you're screwed. It doesn't recover or catch it.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login