Reviewing threat models and security designs has been a large part of my work throughout my career. Increasingly, I come across documents that appear to have been drafted with the help of generative AI. I can often tell within a few seconds. Writing a document has become almost free. So has producing something that looks like reasoning. What has not become free is reasoning that actually holds up. For every design that genuinely works through the problem, that sits with the current state, digs into why it looks the way it does, and only then proposes a solution, I see many more that give me a clean, high-level overview and stop there. The polish is real. The reasoning underneath is thin.
The gap between a document that looks done and one whose assumptions actually hold is where a lot of my job lives now. And there's a useful way to think about what builds up in that gap. I think of it as cognitive debt. It behaves like technical debt: you take a shortcut to ship on time, knowing you'll pay interest later. The difference is that here the shortcut is the thinking itself. When you accept a machine's judgment without doing the reasoning that would let you challenge it, you've borrowed against your own competence. This piece is about how that debt builds, what it quietly costs a security team, and what I think we can do to keep paying it down, before the interest gets away from us.
Why this wave is different
Security has leaned on automation for a long time, and most of it has earned its keep. Tools compiled our code and stood up our infrastructure. They enforced the rules we wrote down. But the division of labor was clear: machines did the executing, and people did the reasoning. We were the ones framing the problem, setting the assumptions, deciding what "secure enough" meant for a given system.
Generative AI blurs that. When a model proposes a trust boundary, or decides which test cases are the ones that matter, it isn't carrying out a decision we already made. It's making the call. And that call comes loaded with assumptions about who the attacker is and how much risk we're willing to accept. Those assumptions used to be ours to set. The older tools bought us time to think about the hard stuff. This wave offers to do the thinking, and that is a far easier thing to say yes to than it should be.
How you get here without ever deciding to
No one sits down one morning and resolves to stop reasoning. It happens gradually, and every step looks reasonable on its own.
Part of it is human nature. The output comes back fast and clean, laid out with a confidence that makes double-checking it feel almost impolite. I've felt the pull myself. It's surprisingly easy to defer to it rather than trust your own read. The effect compounds. Every time it's right, you trust it a little more, and you look a little less hard the next time.
The organizational side is quieter and harder to fix. Technology organizations reward speed, and delivery timelines are often tight. In that environment, the engineer who stops a release to ask whether the AI's threat model actually holds up may not come across as careful. They may come across as the reason the release slipped. So after a while, they stop asking. Run that dynamic across enough sprints and the reflex to question is just gone. Nobody had to ban it; it atrophied.
What it costs
The bill shows up in two different places, and neither sends a warning first. The one that worries me most is what it does to people. Grinding through test coverage, writing the docs, walking a threat model one step at a time: that work is how engineers built judgment in the first place. Hand all of it to the AI and the folks coming up behind you get handed the answer before they've ever had to earn it. They never build the instinct that a seasoned practitioner can't quite explain, the thing that makes the back of your neck prickle before you can point to why. Keep that up long enough and you end up with a team that operates the tools beautifully and has no idea when the tools are lying to them.
The second cost is the enterprise's, and it's simpler. AI is remarkably good at producing work that looks correct. A threat model with nothing left blank. An architecture diagram that reads as airtight. But finished-looking and sound are two different properties, and the distance between them is exactly where the real exposure hides. Pile up enough confident-looking assurances that nobody genuinely worked through, and what you've manufactured is false confidence. That's worse than honest uncertainty, because it feels like safety, right until it isn't.
Agentic systems raise the stakes on all of this. Once the AI is acting instead of just suggesting, fetching data and calling the next service on its own initiative, the reasoning you skipped isn't sitting in a document waiting on a review that might never come. It's already live in production. Whatever judgment you handed off is now running at machine speed, and by the time you go to look, the window to catch a bad assumption has already shut.
Putting judgment back in on purpose
So what do we actually do about it? The answer isn't to use less AI. It's to stop treating human judgment as something a conscientious engineer supplies on a good day, and start building it into the system as a requirement. A handful of practices cover most of it. I care about the first one most, so I'll focus there.
Start by deciding your hard limits and writing them into the system. Every security-critical system has invariants, things that must remain true no matter what a model recommends. Those need to come out of your senior engineers' heads and become policy that's explicit and enforced. An invariant that lives only in someone's memory leaves the building the day they take a job somewhere else. One that's encoded stays, and the machine has to honor it regardless of what it might otherwise have suggested. I'd fight harder to keep this practice than any of the others, because it's the one that survives turnover without depending on anybody.
From there, make every change earn its way past those limits before it ships. "Humans in the loop" can't mean a person reviewing every line. That was never going to scale, and everybody knows it. What it can mean is that the check lives on the road to production. Anything the AI proposes has to demonstrate it doesn't violate your invariants before it goes live, and it's held to the same standard as code somebody wrote by hand. The verification runs on its own, and the bar it has to clear doesn't move.
The last one is to keep asking for evidence while the system runs, not only at review time. Systems drift. Something that was true the day you shipped can quietly stop being true as the things around it change, and nothing fires to tell you. So watch production for signs that an invariant has been crossed, and treat a crossing as something you actually respond to rather than something you note.
What I'd tell a security leader
None of this is an argument against the technology. I use it every day and I'd defend that. What I'm arguing against is a team that trusts its own systems and can't tell you why. If I could hand a security leader two things to do about that, it would be these.
First, set the expectation with every team that the people must reason. They have to know the system, dig into the details, and own the decision. Not just relay the machine's output, but be able to say plainly what they're choosing, what risk they're accepting, and why. Making that call, with a clear view of what it means and what it costs, is the job. Passing along an answer nobody actually decided on is not.
Second, hold the line on your invariants. Those properties that must always stay true, the ones we talked about encoding earlier, are what keep a confident recommendation from quietly overriding a decision your organization already made. Decide what they are, write them down, and enforce them so nothing can move them without someone knowing.
The tools will keep getting better. Whether we can still check them is the part that's up to us.
The views expressed in this article are the author's own and do not necessarily reflect the views of her employer.
