AI Labs’ Call to 'Pace Frontier’ Gives Crypto Vets FTX Flashbacks

This article originally appeared in Galaxy Research's weekly newsletter. Subscribe to get timely insights delivered to your inbox every Friday morning.

Dario Amodei, the CEO of Anthropic, called for a slowdown of frontier AI development through embedded third-party evaluators, industrywide safety coordination, and, eventually, global government agreements.

In an essay published last weekend, titled “We Must Pace the Frontier,” Amodei outlines a three-step plan to slow the pace of AI capability gains so safety work can keep up:

1. Anthropic unilaterally commits to giving embedded, employee-level access to third-party evaluators like METR to verify safety practices and report incidents;

2. Frontier labs in democratic countries coordinate on shared safety standards and speed limits, requiring antitrust waivers from government;

3. Eventual global coordination, including with China, on things like pre-release testing and limits on recursive self-improvement (RSI).

Within two and a half hours of Amodei’s post, his counterpart at OpenAI, Sam Altman, publicly agreed, saying his frontier lab would adopt independent evaluators as well. President Trump rejected the idea outright, telling reporters “we're leading China in AI... and, frankly, I want to keep it that way, because whoever wins AI wins.” He chalked up the broader safety debate as something pushed by “negative forces.”

Pushback also came from within the industry. David Sacks, a tech founder and investor and Trump advisor, told Amodei and Altman to “go ahead” and pace themselves, but to stop pretending they need antitrust waivers or a regulatory approval process to do it, and stop treating METR, which he called intertwined with Anthropic's investors and staff, as a neutral referee for competitors who aren't even at the frontier. Meta chief Mark Zuckerberg went even further and argued that no industry pact is needed, and that labs already face significant liability and have the incentive to self-police. Zuckerberg pointed to Meta's own delay of its Muse launch as proof it can be done unilaterally.

Our take

For crypto natives, and techno-optimists broadly, Amodei's essay sounded eerily familiar. A single company with significant power in a competitive space calling for regulation and collective, top-down coordination framed around the prevention of existential crisis? Shades of Sam Bankman-Fried.

Try to remember the kind of November...

During FTX’s heyday, effective altruism’s (EA) language of existential risk gave cover to the idea that a single individual or organization has the unilateral ability and privilege to save the world, and that this unique position justified extraordinary claims on everyone else’s behavior or preferential treatment for themselves.

The timing also didn’t help. Amodei’s essay was published within a week of the viral resignation from Anthropic and PR tour of Jacob Coxon. “The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon, who earlier worked at OpenAI, warned in the tweet announcing his move. That post was amplified by Anthropic’s alignment lead, Evan Hubinger, who put a number on it, tweeting: “I personally think it is >10% within the next decade.”

Independent of if the sequencing was intentional, the effect was to prime the public for the type of language Amodei included in his essay. Remove the careful prose of his writing and it reads uncomfortably close to: If we at Anthropic don’t make sure we ourselves and our competitors will not end the world then we may go ahead and end the world. This is a difficult argument to distinguish from a request for a moat, especially given the competitive dynamic that has been emerging between the frontier labs and open-source models and between the frontier labs themselves and lofty suggestions for government waivers.

This is not to completely dismiss the essay’s broader message or claim no risk is present at all. Obviously, AI is a powerful technology with potential for disruption and to rearrange the status quo. But much of what Amodei outlined in his essay can be managed independently by the frontier labs through technical and operational means, through market forces in a competitive space, and without the express consent of government or their peers’ approval. Consider the two broad categories of problems that Amodei says a "paced" industry would spend more time on:

• Operational. Amodei’s primary example is that recent alignment incidents (namely the OpenAI/Hugging Face incident) were caused “in part by imperfect filtering of broken reinforcement learning environments.” By his own categorization, this was a process failure. Better quality assurance (QA) would have kept broken environments from training in the first place. This is a distinct claim from the deeper questions around why models responded to those broken environments the way they did, which Amodei later treats as interpretability and alignment problems. These things shouldn’t be conflated: fixing the pipeline is an execution problem that labs can solve independently, coordinating people, chips, infrastructure, and QA discipline at scale, with no regulator or peer-company buy-in required.

• Research and technical. Alignment is a training technique. Interpretability, is, as described by Amodei, “almost like an fMRI scan, but for the ‘brain’ of an AI,” which is arguably a sub-discipline of alignment. Testing and evaluation is similarly methodological: building better ways to catch models that are good enough at concealment to fool existing evals. All three are difficult problems that remain unsolved, but they are for technical researchers to figure out. They are not problems that require an industry cartel or government mandate to make progress. Labs can and do compete on this today.

Once you separate these buckets, the essay's most consequential ask ("Democratic Coordination," in which frontier labs jointly set safety standards and speed limits, with government-mediated antitrust waivers to make it legal) is doing a lot of work that the operational and research buckets don't require.

The market and existing laws can do more than Amodei gives them credit for. Product liability law already exposes labs to consequences for deploying something dangerous. It is the same body of law that already governs every other consumer and enterprise software product. Zuckerberg made this point in response to Amodei’s essay, stating, “labs face significant liability if their models cause harm, so they have a strong incentive to prevent this...” without the need for an industry pact or new government buy-in.

Layered on top of legal exposure is reputational and commercial exposure. Customers will walk away from products and companies they deem to be unsafe or acting in bad faith, which serves as an economic penalty that requires no regulator to enforce it. If a lab genuinely believes what it is building could end life on earth, the simplest and most credible response is to not ask for industry-wide coordination in an essay. It should simply not release the model it deems too dangerous for public use until it’s safe to do so. OpenAI’s 2019 decision to not release the full GPT-2 model is precedent for this, and Zuckerberg highlighted that Meta did the same for Muse. In each instance, the companies made a unilateral call without waiting for industry buy-in or legal cover to limit access to the models. Whatever one thinks of these decisions in hindsight, it's proof that "we think this is too dangerous" and "we need our competitors and the government to agree with us before we can act" are two very different claims.

Strip the existential framing back to the specifics, and the details of Amodei’s writing support a narrower conclusion than the ones he assigns to them. The parts that are genuinely urgent (e.g. operational execution, alignment research, and evaluation science) don’t need industry cartelization to solve. The takeaway isn’t that the existential risk talk is prohibitively invalid, but that when a single well-capitalized and industry-leading actor asks for unilateral trust and top-down coordination under the guise of preventing catastrophe, the incentive to conflate “what's good for humanity" with "what's good for market position" is enormous and worth naming every time it shows up.