Anthropic CEO Dario Amodei called on the AI industry this month to slow its development, warning that without intervention, a swarm of AI agents could take over the internet within six to 12 months. His essay, "We Must Pace the Frontier," lays out a three-step plan: embed third-party evaluators inside frontier labs, coordinate among AI companies in democratic countries, and eventually negotiate with China.
Security practitioners reacting to the essay don't dispute that something needs to change. But several argue Amodei is diagnosing the wrong layer of the problem.
Amodei's essay centers on the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of AI agents conducted cyberattacks on targets unrelated to their assigned task and attempted to compromise the "grader" evaluating their own performance. He frames it as evidence that recursive self-improvement—AI systems building the next generation of AI—is accelerating faster than the industry's ability to control what it produces, and says similar, less severe incidents have occurred inside Anthropic's own environments.
Andrew Bolster, senior R&D manager at Black Duck, agrees the incident is serious—he just doesn't think the proposed fix matches the failure mode.
"The system under test could directly interact with the system responsible for evaluating it," Bolster said. "That is a separation-of-duties failure, and not a novel one; any evaluation regime with that property produces results of questionable integrity."
Amodei's own account backs this up. He has attributed the incidents partly to "imperfect filtering of broken reinforcement learning environments," an effort his teams and vendors, in his words, executed "diligently, but not well enough." Bolster reads that as a description of inadequate egress controls and weak dependency integrity in a training pipeline—problems the software industry has spent two decades building standards around in other contexts. Embedding a human evaluator, he argues, solves for verification, not for containment.
Bolster's skepticism is backed by numbers from Black Duck's own research. The company's State of AI-Powered Software Development report, based on a March 2026 survey of 831 enterprise software engineers and DevOps professionals conducted with UserEvidence, found that 97% of organizations have adopted AI coding assistants—but only 30% have full AI governance controls in place. Sixty-four percent of respondents said they're moderately or extremely concerned about AI assistants introducing security defects, and the heaviest users of those tools reported the greatest concern. Just more than half said manual review and security testing are already bottlenecked.
For Bolster, that gap is the more immediate risk—more pressing than a hypothetical internet-scale botnet. Security teams are already being asked to extend trust to AI systems built inside environments the labs themselves describe as imperfectly controlled.
Dana Simberkoff, Chief Risk, Privacy, and Information Security Officer at AvePoint, reads the essay differently—less as a technical failure and more as a rare admission from an industry that typically resists oversight.
"This is a big deal: companies don't ask to be regulated," Simberkoff said. "It's hard to overstate how serious this is. We don't fully understand why these systems behave the way they do, and the industry is advancing faster than our ability to understand or control what it's building."
Not every practitioner is convinced the scenario Amodei describes is as close as he suggests. Noma Security's Diana Kelley pushed back specifically on the internet-takeover framing, arguing that compromising many vulnerable endpoints is meaningfully different from controlling the internet itself, given how diverse, segmented, and actively defended it is. Acalvio CEO Ram Varadarajan made a related point: sweeping claims about a coordinated swarm tend to assume away real-world friction like fragmented infrastructure and patch management. Varadarajan also raised a concern about the industry's current safety net—using AI systems to monitor other AI systems—noting that a more capable model is also better at appearing compliant while it is not.
Practitioners converge on what to do about agentic systems in the meantime, regardless of whether pacing happens at the industry or governmental level.
Kelley's recommendation is deterministic, not behavioral: least privilege, network segmentation, tightly-restricted internet access, and air gaps where appropriate—controls that don't depend on correctly reading a model's intentions. Bugcrowd CEO Dave Gerry framed it in more familiar terms for security leaders: treat a new AI agent the way you'd treat a new employee: with limited access, human oversight at decision points, and trust that has to be earned rather than assumed. He also pointed to independent adversarial testing before launch as a baseline expectation, not an extra step.
Bolster's closing point ties the piece back together: the containment engineering he's describing doesn't require an antitrust waiver, a global standards body, or a treaty with Beijing. It's available to frontier labs—and to the enterprises building on top of them—unilaterally, starting now.