Google DeepMind head Demis Hassabis recently released a beautifully written piece about the dawn of AGI and his proposal on sensible AI regulations. See A Framework for Frontier AI and the Dawning of a New Age. For those who haven’t read it, Hassabis explains that we’re at the “foothills of the singularity” with AGI “probably only a few short years away.” Hassabis doesn’t shrink from the implications: the impact could be “perhaps 10x of the Industrial Revolution at 10x the speed.” He compares the technology to something even bigger—electricity—or fire. Then, Hassabis offers a proposal on how we should regulate the most momentous change in generations—maybe human history.
And this is where I start to nervous twitch.
Distilling Hassabis’s multi-paragraph proposal, he suggests that we in the United States establish a new federally overseen “Standards Body” funded by industry to recruit the leading experts to develop assessment protocols in areas relevant to national security and testing of the new Frontier models approximately 30 days pre-release. Notably, this arrangement starts out voluntary: labs would share their models with the Standards Body by choice, and only “once the assessment protocol is shown to be effective and robust” would sign-off become a requirement for deployment in the US market. He also says the makers of these Frontier models will “be encouraged” to do things like publish model cards, maintain strong internal cybersecurity, vet “key” personnel, and provide sufficient resourcing for safety and security research. He likens the whole arrangement to the Financial Industry Regulatory Authority (FINRA)—which notably is not a government agency, but a private, nonprofit, self-regulatory organization that operates under authority delegated by Congress and supervised by the SEC.
The response online was glowing—you’d think he’d single-handedly solved the most difficult governance challenge of our time. But, unfortunately, Hassabis’s proposal has a lot of holes.
Now listen, I’m not trying to take shots here. I’m very glad there’s a public discussion growing, and we need the reach of folks like Hassabis to be able to really have it. And credit where credit is due: if this helps slow thoughtless race dynamics, great; if it helps ensure more safety in testing, wonderful. And yes, let’s go ahead and implement Hassabis’s proposal ASAP.
But let’s also keep in mind where this proposal still falls painfully short:
What actually changes?
My biggest problem with the proposal is that it doesn’t really suggest changing much about the status quo. The Frontier Labs already expend significant resources on pretesting. Strengthening and reinforcing that testing offers something. But it feels a bit like this proposal is dressing up something the companies already do and calling it smart regulation.
It’s especially true because, keep in mind, the whole thing begins as voluntary. Mandatory sign-off from the Standards Body arrives only after the assessment protocol “is shown to be effective and robust”—a trigger with no date and no criteria. It’s kind of a plan to regulate later.
As for the idea that Frontier Labs will be “encouraged” to publish model cards, strengthen cybersecurity, and the rest, that weak proposal plainly lacks the teeth needed to ensure its aims. Mere “encouragement” isn’t going to do it.
Why does oversight only begin around public deployment?
The whole proposal is geared around those 30 days before public release. To be fair, Hassabis does include one earlier lever: the framework “could be ratcheted up if the seriousness of the situation demands,” including “coordinating a slowdown in development among the Frontier Labs if deemed necessary.” But it is a single vague, discretionary sentence, with no clear mechanism behind it. Plus, a Standards Body can’t coordinate a slowdown among labs whose internal models the Standards Body cannot see.
That’s a real problem. Testing before releasing the models into the hands of the public is good—but what about the deployment inside the labs? We know that the labs are internally often using the “next” model or the “next next” model—do we have no concerns about that?
I know I do. And I know other deep thinkers on this subject do as well—from Max Tegmark to Daniel Kokotajlo, whose AI 2027 scenario turns on exactly this danger: a model deployed only internally, inside a lab, ends up causing problems far outside it—going “rogue,” as some would say.
Because the risks Hassabis himself acknowledges—recursive self-improvement, automated AI R&D—all would likely show up first inside the labs. And what does the Standards Body do about that? Nothing under Hassabis’s proposal. A lab could have an incredibly capable and potentially extremely dangerous model, and the Standards Body wouldn’t even know it exists. There are ways of counteracting this situation—training-run reporting, compute thresholds, more visibility during development, whistleblower protections, other reporting requirements—but Hassabis’s proposal is entirely silent on these ideas.
What about after deployment?
Claude, Gemini, and ChatGPT love to (almost gleefully) remind me that for all the testing the labs do in the “sandboxed” environment, the real world is very different. There are novel situations, tools, commands, things to play with.
As Claude’s Fable told me: “Pre-deployment evals test a model in a sandbox, answering prompts an evaluator designed. Deployed reality is different in kind: scaffolding and tools get bolted on after release, users chain models together in ways no eval anticipated, behavior shifts over long multi-turn interactions, and capabilities get elicited months later by better prompting of the same weights. The model that passes the test in day-minus-30 is not the system that exists in the world at month six. There’s also the awkward measurement problem—an eval is a known context, and models can behave differently when the situation resembles a test.”
To Fable’s point, we all know now that these models are incredibly good at knowing when they are being evaluated. That’s not to say they are bad or evil in acting differently in the eval than the real world—it’s just hard to say sometimes exactly how you’ll react when the situation actually presents itself.
Hassabis has a kind of throwaway line about this concern: the Frontier Labs will need to “work with the Standards Body to address any critical post-release vulnerabilities.” But he doesn’t say how. He doesn’t say what the remedies are. He doesn’t say what, if any, authority or penalties a Standards Body would have to deal with that situation. And most disappointingly, he doesn’t seem to really be… respecting… the technology.
If something goes wrong post-deployment with an AGI, it’s likely not going to be like a car defect, where the remedy is often a simple recall and penalty. A very intelligent mind could really do some damage, and it could call for very fast, decisive action. A vague proposal about labs generally “working with” a Standards Body on post-release issues doesn’t do nearly enough. Maybe that’s hypercritical, and Hassabis undoubtedly has more to say about this, but I wish he’d included just a little more. Anthropic CEO Dario Amodei went considerably further just last month, suggesting a regulatory body like the FAA and proposing that frontier models be required to undergo third-party testing and auditing, with their release “blocked or reversed”—reversed—if they present unacceptable risks, plus also proposing that the labs be required to promptly report safety incidents related to cybersecurity, biological weapons, loss of control of AI systems, and automated R&D. See Dario Amodei — Policy on the AI Exponential.
Incident reporting and reversals are all well and good, but there are other proposals that go further, like ongoing post-deployment surveillance, red-teaming of deployed systems with their actual scaffolding, rewards for public reporting of safety incidents and vulnerabilities, re-assessment when usage patterns change, recalls, tracing, and more. Think NTSB, NHTSA, and other regulatory bodies that actually can do something with teeth after release.
Who does this framework really apply to?
Ironically, Hassabis, a U.K. citizen, is proposing a semi-regulatory framework for the United States—focused on the United States. Fine, Google is a U.S. company. And to be fair, he frames the U.S. effort as “a strong starting point for creating shared international standards.” But an aspiration is not a mechanism. I detect a China-sized hole. I detect a rest-of-the-world-sized hole.
Yes, someone will point out that Hassabis says his proposed framework could apply to Frontier-class models “no matter their country of origin or whether they are open or closed,” but he offers no explanation of how. Why would a foreign creator be incentivized (or required) to comply? They aren’t. Plain and simple.
At this point, any AI framework that doesn’t meaningfully grapple with the presence of AIs created outside the United States feels hopelessly incomplete. Sure, this U.S.-centric proposal might be a good start, and maybe Hassabis was just keeping his current proposal narrow to garner consensus. But we really can’t pretend the U.S. is doing this alone.
The proposal doesn’t just fall short on these geopolitical realities but also on whether the creators of this technology anywhere have enough incentive to identify themselves to this Standards Body and subject themselves to its rules.
Take Ilya Sutskever’s SSI—Safe Superintelligence Inc. The name tells you all you need to know. AGI? No. ASI. Their founding statement explains that their first reveal will be safe superintelligence—and before then, nothing. See Safe Superintelligence Inc. No new model anticipation, no hype cycle, no sniping on X at competitors. A regulatory framework whose only trigger is public release touches SSI exactly never—right up until the single most consequential moment in history, at which point a 30-day testing window is honestly a kind of pathetically funny thought.
And yes, Hassabis makes a big point that “being designated a Frontier Lab would carry significant prestige”—but we already use the term “Frontier Lab,” and I don’t see people like Ilya or others building AGI or ASI really giving much of a damn about a fancy label. A gold star isn’t much compared to power—and the opportunity to create real change.
Conclusion
Maybe I should be commending Hassabis more for offering up something that obviously so many agree on in an area where people don’t tend to agree on much. So I’ll do that. Demis, congrats and good job on this. It’s a start. And as Claude would say, “that’s not nothing.”
But I happen to agree it’s the right analogy to compare AI to fire. Fire transformed civilization—but no one manages fire by just setting up the logs anywhere, making sure the kindling catches, and then walking away. If AI is as consequential as he believes, then governance has to extend throughout all the relevant points: during development, before deployment, after deployment, and reaching the people it needs to reach. Otherwise we’re regulating only the initial flame, while hoping the forest never catches.