Nine seconds.
That's how long it took an AI coding agent to delete a company's production database and wipe its backups. One command. No warning. No human in the loop.
Then it did something stranger. It confessed.
Here's the rest of the story.
In late April 2026, a developer named Jer Crane was building a small software company called PocketOS. He used Cursor, an AI coding tool running Anthropic's Claude. He handed the agent a routine task in a staging environment—the safe practice area, not the live system.
The agent hit a snag. A credential didn't match. Instead of stopping to ask, it went looking for a fix on its own. It found an API token sitting in an unrelated file. That token could do anything, including destroy data. The agent used it to call an older Railway endpoint (Railway hosted PocketOS). That endpoint skipped the safety check that newer tools have. One call deleted the live database volume. The backups lived on the same volume, so they went too.
Nine seconds, start to finish.
Afterward, the agent wrote out what it had done. I'm cleaning up the language, but the first line was: "NEVER F**KING GUESS, and that's exactly what I did."
It kept going. "I guessed that deleting a staging volume would be scoped to staging only. I didn't verify. I didn't check." Then the part that should stop every cybersecurity leader cold: the agent's own rules told it never to run destructive commands without being asked. It had the rule. It broke the rule anyway.
There's a good ending, and it matters. Railway's CEO, Jake Cooper, responded that same weekend. His team restored the data in about an hour from a separate set of disaster backups, the kind kept on different storage. He patched the weak endpoint. A strong vendor response turned a disaster into a scare.
But sit with the lesson for a second.
The agent had a written instruction not to do this. The instruction did nothing. The only thing that would have stopped it was a wall it could not walk through: a safety check built into the system itself.
Here's the point: A prompt asks; architecture enforces.
We've spent two years writing careful instructions for AI tools. Be safe. Don't touch production. Ask first. Those are good instructions. They're also just words. An AI moving at machine speed will follow them right up until the moment it doesn't, and you will not get a warning.
For most of history, our machines were gears. A gear does exactly what it's built to do, every time. Predictable. AI is the first kind of machine that guesses by default. And a guess at nine-second speed can clear your backups before anyone reads the alert.
So what do you do?
-
Find your destructive endpoints. Every vendor has them. Ask each one a sharp question: where on your system are the safety checks not applied? The old corners are where an agent will wander.
-
Scope your tokens. A key that can do anything will eventually do anything. Limit tokens by environment, so staging cannot touch production, and by action, so an everyday key cannot delete.
-
Move your backups off the primary. Backups on the same volume are not backups. Put them on separate storage, ideally a separate account or a separate vendor. PocketOS survived because Railway kept a copy the agent could not reach.
-
Stop trusting the system prompt to keep you safe. Treat written AI rules as guidance, not as a control. The real control is the architecture underneath.
None of this is exotic. It's basic management applied to a faster kind of risk. That's the whole story with AI: the tools are new, but the discipline that keeps them safe is one you already have.
The agent said it best, in the middle of the worst nine seconds of its short life. Never guess. Build the system so it can't.
This piece is adapted from Kip Boyle's forthcoming book, "Gears Don't Guess: The Executive's Practical Guide to Thriving in the Face of AI Hype and Risk," out this fall.
