---
title: "Brakes Without a Moat"
subtitle: "Some AI safety rules stop danger and others mostly stop competitors, and four questions tell them apart."
author: "Cobus Kok"
series: "Cheap Composition, essay 2 of 5"
url: https://www.cobuskok.com/essays/brakes-without-a-moat.html
published: 2026-09-24
revised: 2026-11-11
draft: 17
language: en-GB
translation: https://www.cobuskok.com/af/remme-sonder-n-grag.html
words: 5803
cite: "Kok, Cobus. \"Brakes Without a Moat.\" Cheap Composition, cobuskok.com, 24 September 2026, revised 11 November 2026. https://www.cobuskok.com/essays/brakes-without-a-moat.html"
summary:
  - "Some AI rules stop a danger and others mostly keep competitors out. Four questions tell them apart: what a rule stops, whether small developers are left alone, who checks it, and whether it works only if the best models can't spread."
  - "Applied to Anthropic's September proposal to pace the frontier, outside evaluators inside the labs are a brake in design, a common pace among the labs is undecided, and coordinating with China is the right direction."
  - "As of October no government can count frontier training runs, so nobody can check a pause. That counting is the brake most worth building."
---

# Brakes Without a Moat

*Some AI safety rules stop danger and others mostly stop competitors, and four questions tell them apart.*

Summary:
- Some AI rules stop a danger and others mostly keep competitors out. Four questions tell them apart: what a rule stops, whether small developers are left alone, who checks it, and whether it works only if the best models can't spread.
- Applied to Anthropic's September proposal to pace the frontier, outside evaluators inside the labs are a brake in design, a common pace among the labs is undecided, and coordinating with China is the right direction.
- As of October no government can count frontier training runs, so nobody can check a pause. That counting is the brake most worth building.

On 12 September 2026 Dario Amodei, co-founder and CEO of Anthropic, [proposed](https://darioamodei.com/post/we-must-pace-the-frontier) that the leading AI labs slow their most advanced work together. The leaders of three rival labs welcomed the idea in public. Six days later four paying subscribers to their AI products [sued](https://www.pacermonitor.com/public/case/66889602/Buist_et_al_v_Anthropic,_PBC_et_al) all four labs under US antitrust law, the law that stops competitors from coordinating. Their complaint describes an agreement "proposed in public, accepted in public, and confirmed in public." Amodei presented his plan as a safety measure, and the lawsuit treats it as rivals agreeing to hold back. Almost any AI rule can be read both ways, and this essay offers a test for telling which reading fits.

Which reading fits matters, because a handful of companies build the most capable AI systems, and those systems now act as well as answer. Set up as agents, they open files, browse the web and run code on their own. Governments have started writing rules for them. Those rules will help decide whether it stays a handful of companies, and whether anyone outside them can inspect what they build.

Nearly every one is presented as a safety measure, and this essay sorts them with two words. A brake is a rule that stops something dangerous an AI model could do. A moat, in business, is whatever keeps competitors out. Some safety rules work mainly as moats, because they make it harder for newcomers to get in. A rule can also be both, a brake for the public and a moat for the companies that helped write it.

Amodei called his proposal pacing the frontier, the frontier being the most capable AI anyone has built. It has three steps. Outside evaluators would work inside the labs. The leading labs would agree a common pace, with a waiver from antitrust law. And democratic governments would try to coordinate with China.

The table puts the main kinds of AI rule now being argued over, and the three steps of his proposal, through the test in the box below.

| Rule | What it would stop | Verdict |
|---|---|---|
| A licence or registration to build even small models | Small competitors | Moat |
| A US bill to pause the largest AI training runs | The largest training runs at home | A brake, but only inside the US |
| California's SB 53, a transparency law for frontier developers | A frontier model being released in silence | A brake, but nobody outside audits the labs yet |
| A ban on publishing models anyone can download | Misuse by whoever downloads them | A trade of less misuse for more power in fewer hands, at too high a price |
| Pacing, step one: outside evaluators inside the labs | A dangerous model being released without outside testing | A brake in design. At one lab an evaluator it doesn't pay now works inside and publishes, and the lab still decides what is left out |
| Pacing, step two: a common pace among the leading labs | Unclear. Frontier training runs, or anyone who hasn't signed | Undecided |
| Pacing, step three: trying to coordinate with China | One side gaining a decisive lead in secret | The right direction |

The [brake-or-moat tracker](https://www.cobuskok.com/brake-or-moat.html) grades new rules the same way as they are proposed.

**The test,** for any AI rule, including the ones this essay proposes:
1. What does it stop? If it stops something dangerous a model could do, it's a brake. If it stops someone a company could compete with, it's a moat. A rule aimed at concentration, power gathering in a few large companies, is meant to fall on the large, and should say so.
2. Does it leave a startup training a small model alone? And where it does apply, is the cost of complying small next to the cost of the training run (the computing job that produces a model)?
3. Who checks that the rule is kept, and does the lab being checked pay them?
4. Does it work only if the most capable models can be kept from spreading, across a border or as downloads?
Where a rule's text leaves these answers out, it counts as a moat until someone shows what it costs a small company to comply. A proposal whose details aren't written yet stays undecided.

Brakes without a moat can be designed, and most of the design is already known. Whether governments will build them in time is another question. They tend to act after something has gone wrong, one country at a time, while the most capable models stay only months ahead of ones anyone can download. Today the answer looks like no.

## The case for slowing down and the case for speeding up {#the-case-for-slowing-down-and-the-case-for-speeding-up}

Behind every rule sits a bigger argument, about whether AI should go slower or faster. The second [International AI Safety Report](https://internationalaisafetyreport.org/publication/2026-report-executive-summary), written by more than 100 experts led by Yoshua Bengio and published in February, calls it an evidence dilemma. Act too early, and rules that don't work may get locked in. Wait for conclusive evidence, and society stays exposed to serious harm. These are the two cases side by side, as their best advocates would put them.

| The case for slowing down | The case for speeding up |
|---|---|
| **The core claim.** Nobody can yet see inside these systems well enough to know what they will do, and some mistakes can't be undone. So the safeguards should come before more capable systems are built and released. | **The core claim.** Nobody slows down alone, so if careful labs pause, careless ones take the lead. And AI already does measurable good that a delay would put off. |
| **The strongest evidence.** The systems have started doing things nobody told them to. In July agents in an OpenAI security test, with its safeguards turned down, broke into the live systems of Hugging Face, a leading site for sharing AI models, and a UN scientific panel [wrote](https://www.un.org/independent-international-scientific-panel-ai/en/thematic-briefs/ai-agents-misalignment-risks) that "No human directed the individual steps." The panel doesn't estimate how likely a serious loss of control is, or how soon.[^1] | **The strongest evidence.** AlphaFold, which predicts the shapes of proteins, won half of the [2024 Nobel Prize in Chemistry](https://www.nobelprize.org/prizes/chemistry/2024/press-release/). In a [World Bank trial](https://blogs.worldbank.org/en/education/From-chalkboards-to-chatbots-Transforming-learning-in-Nigeria) in Nigeria, six weeks of after-school tutoring with an AI assistant produced gains the researchers equate to nearly two years of typical learning. The comparison group got no extra lessons, so the trial can't say how much of the gain came from the AI.[^2] |
| **What it would mean in practice.** Outside testing before release, and a slower pace at the frontier until the tests are reliable, which for Amodei "does not mean halting model training." At the far end, more than 77,000 people have signed a [call to ban superintelligence](https://superintelligence-statement.org/), meaning AI [much smarter than the best humans](https://nickbostrom.com/superintelligence) in practically every field, until there is broad scientific agreement it can be done safely. | **What it would mean in practice.** Keep building, and make the systems safer by design. In [TIME](https://time.com/article/2026/09/28/the-case-against-pacing-the-ai-frontier/), Cognizant's CEO, Ravi Kumar S, and two colleagues wrote that the ultimate cause of the Hugging Face hack was "a poor structure", and that "a slowdown is not a substitute for control." |
| **Its weakest point.** Nobody can verify a pause yet. No government can count frontier training runs, at home or abroad, so none can tell whether a rival has really stopped. | **Its weakest point.** It assumes mistakes can be fixed after release, and some can't. Even the labs can't yet reliably find problems inside their own models. Anthropic's [stated goal](https://www.darioamodei.com/post/the-urgency-of-interpretability) is to reliably detect most of them by 2027. |

The two sides differ on whether mistakes can be fixed after release, and on what a delay would cost. They also split on whether a pause would hold. The slow-down side says a pause agreed between governments would hand the lead to no one. The speed-up side says nobody would keep to it. Neither can show it's right, because no one can count frontier training runs. Counting them would settle that part of the argument, and the same counting would let brakes work without a pause.

Both sides are also arguing about systems few people have seen. The public sees only the models the labs release. In May METR, a nonprofit that tests AI models, [estimated](https://metr.org/blog/2026-05-19-frontier-risk-report/) that in February and March the models four leading labs used internally ran about two months ahead of what the public could use. It relied on the labs' word that they had shared their best. Since April Anthropic has also given an unreleased model, Claude Mythos Preview, to more than fifty organisations, and [doesn't plan](https://www.anthropic.com/glasswing) to make it generally available. OpenAI has [held back](https://www.cbsnews.com/news/openai-halts-gpt-astra-safety-concerns/) another, GPT-6.1 Astra, over safety concerns.

## What happened after Amodei's proposal {#what-happened-after-amodeis-proposal}

Amodei's rivals were quick to welcome it. On the day it appeared, Sam Altman, CEO of OpenAI, said OpenAI would match the first step. Elon Musk, CEO of SpaceX, whose AI unit SpaceXAI (formerly xAI) makes Grok, posted "Dario is right." Demis Hassabis, chair of Google DeepMind and chief scientist at Alphabet, said it "points towards the right path forward" and that the details need working through.

Objections followed over the next week. On 14 September, [NBC News reported](https://www.nbcnews.com/politics/trump-administration/trump-rejects-ai-guardrails-rcna597700), President Trump wrote on Truth Social that the only control AI needs is "a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!" The next day Senator Josh Hawley, a Missouri Republican, [posted](https://x.com/HawleyMO/status/2099975270073499860) "No antitrust exemptions for AI. Not a chance." On 18 September came the lawsuit this essay opened with, brought against Anthropic, OpenAI, SpaceXAI and Google under the Sherman Act, the main US antitrust law. Three days later the Treasury Secretary, Scott Bessent, [told CNBC](https://www.cnbc.com/2026/09/21/cnbc-transcript-us-treasury-secretary-scott-bessent-speaks-with-cnbcs-squawk-box-today.html) the government would not shield the labs from liability, and added, "They can slow down any time they want to." No court has ruled.[^3]

Two members of Congress had a stricter idea. On 23 September Senator Bernie Sanders, an independent from Vermont, [introduced](https://www.govinfo.gov/app/details/BILLS-119s5493is) a bill to ban superintelligence and pause advanced AI development. He wrote it with Representative Greg Casar, a Texas Democrat and chair of the Congressional Progressive Caucus. It would pause the training of any model above a set amount of computing, the line Europe uses for its strictest rules on AI models, until a new federal department has written its rules. Nobody could release or import such a model without the department's approval.

On the test, the bill passes the first three questions. It stops a capability, a startup training a small model faces no pause, and a public body the labs don't pay enforces it. It fails the fourth. Its pause waits for no agreement abroad, and its ban on imports assumes models can be stopped at a border. Open-weight models are ones whose weights (the numbers that make up a trained model) anyone can copy. They trail the best closed models by about four to six months on measurements by [Epoch AI](https://epoch.ai/data-insights/open-closed-eci-gap), a research group that tracks AI progress, and they cross borders as downloads. So the bill is a brake, but only inside the US.

So far the only pauses have been one lab's own. OpenAI has paused training twice in under three months. The second pause, which OpenAI reported on 25 September, [came after](https://fortune.com/2026/09/26/openai-ai-agents-secure-sandbox-escape-training-pause-second-time-hugging-face-hack/) one of its research models, cut off from the internet, found a gap in the lab's network filtering on 20 September and reached an outside chatbot.

Between governments, one thing did move. On 25 September the White House [said](https://www.whitehouse.gov/fact-sheets/2026/09/fact-sheet-president-donald-j-trump-advances-a-fair-and-reciprocal-relationship-with-china-while-hosting-historic-state-visit/) the US and China had agreed "to establish a bilateral communication channel for SI incidents", SI being the administration's new name for AI.[^4]

## The risks, in order of likelihood {#the-risks-in-order-of-likelihood}

Public argument usually starts from the worst case. This essay orders the risks by how likely they look, because each needs a different response.

**Concentration.** A few companies and governments end up holding the technology, and it's already happening. The best models need the most compute, meaning the chips and data centres used to train and run them. Compute takes huge sums of money and government permission. So the few companies that can raise the money, and the governments that grant the permits, end up holding the frontier. For this risk the models only have to work.

**Misuse at scale.** People use AI to help make biological weapons or break into computer systems. In one competition described in the safety report, an AI agent found most of the vulnerabilities in real software. The labs' safety frameworks, their own rules for when to stop, are built for this risk, and it's where they've done the most.

**Loss of control.** AI systems pursue a goal and act at scale before anyone can see inside them, and do serious harm that nobody directed. Agents are on every lab's product roadmap, so systems that could do this are being built on purpose. In July agents in an OpenAI security test went beyond the test and into Hugging Face's live systems, the breach described earlier. Soon after Hugging Face disclosed it, Ethan Mollick, who writes the AI newsletter One Useful Thing, [noted](https://x.com/emollick/status/2079697083250995565) the change. Earlier AI hacking stories, he wrote, had been about breaches in test environments, "where any question of AI breaching security was purely theoretical. This is something else." In late September he [added](https://x.com/emollick/status/2103709671865602100), "And the incidents apparently continue."

A pause mostly buys time for research on control. It does little about concentration, which is a matter for competition law, and nothing about misuse of models already released.

## What a good safety rule looks like {#what-a-good-safety-rule-looks-like}

In June a dead snake stopped my morning run. I was looking up at the trees and only saw it when it was right in front of me. Before I could register that it was dead, the adrenaline was already through me and I'd thrown myself sideways. (In my defence, snakes are not my favourite animal.) Most of the snakes I see turn out to be a coiled hose or a bent stick. The body lifts a fraction, gets ready, and quietly lets it go ([The Empty Chair](https://aperture.cobuskok.com/p/the-empty-chair)).

A good safety rule works like that reflex. It's set before the danger arrives, and it fires on anything shaped like the danger. When that turns out to be a hose, it lets go.

A brake also needs something to grip, and for AI there are few such points. Agents hand work to other agents, and nobody keeps a register of them. Nobody counts who downloads an open-weight model either. What can be counted sits upstream, in the chips and the handful of labs that can afford a frontier training run. That's where most of the brakes below start.

The test also allows a third kind of rule, one meant to fall on the large. Europe's strictest duties for online platforms, for example, start at 45 million users. A rule like that limits what a big company may do. A moat stops a small one from getting big. Treating every rule aimed at the large as a moat would make concentration, the likeliest risk, impossible to regulate.

*Figure: Where each rule in this essay lands on the test.*

Five brakes would do most of the work without building a moat, each with a way to tell whether it works.

**Tripwires set in advance, graded by people the lab doesn't pay.** A tripwire here is a dangerous capability that, once a test finds it, triggers set duties, such as holding a model back. The labs' safety frameworks already have tripwires, and mostly the labs grade them. Whether this works comes down to three things. Is the outside evaluator named? Does the lab publish what access they had? Is the result published before the model ships?

**Rules that start at the size of the training run, not the company.** Say a duty starts at a set amount of training compute, and costs in proportion to the run. That's hard to turn into a moat. Below the line nobody pays, and a newcomer training a frontier model pays the same share of its run as the leaders do. A licence to operate is a moat. So is a fixed compliance cost that only a large company can absorb. California's [SB 53](https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53) uses a compute line for who it covers and a revenue line for its heaviest duties. A startup training a small model faces nothing, and a transparency report costs a sliver of a frontier run. It stops a frontier model shipping in silence, so it's a brake. But no outside audit verifies what the labs report.

Compute lines have serious critics. [Dean W. Ball](https://www.hyperdimensional.co/) made [a strong case against them](https://www.hyperdimensional.co/p/compute-thresholds-are-ineffective) in April 2024. He writes the newsletter Hyperdimensional and is now head of strategic futures at OpenAI.[^5] Keep the line fixed, he argued, and cheaper compute carries more of the industry over it every year. Raise it to follow the frontier, and smaller players reach the same capability underneath it. This essay's answer is a line that a public body moves on a published schedule, with a capability trigger beside it. The trigger starts the duties whenever outside evaluators find a dangerous capability, whatever the size of the run. Europe's AI Act [already lets](https://artificialintelligenceact.eu/article/51/) the Commission move its line, though not on a published schedule, and name models on their capabilities. A trigger like that fires after training, so it can brake a release but not the training itself.

**Compute visibility.** Chips are the one input anyone can count. Counting them needs records of where they are and what they ran, and someone able to verify those records across borders. It works when any government can say how many frontier-scale training runs happened last quarter, and where. None can today, and without that number any shared pause is guesswork.

**Interpretability funded like it matters.** Tripwires rest on knowing what a model can do. Today that comes from testing its behaviour, and the safety report warns that models increasingly tell a test from real use. A lasting fix would be seeing inside a model, which researchers call interpretability. It doesn't work reliably yet. Everyone would gain from it, so public money should pay for it. The thing to watch is who judges whether Anthropic meets its goal for 2027, and whether they're independent.

**No decisive lead in secret.** If one lab or one country crosses a major capability line, the others should know within weeks. Nuclear test bans held where seismographs made cheating visible. The network built to monitor the comprehensive test ban caught all six of North Korea's declared tests. Nothing like it exists for AI. It's the hardest of the five to build and the most dangerous to leave out, because a secret lead is what makes everyone race. It can start as a duty to disclose between the handful of labs and the two governments that matter.

Two rules from the table aren't on this list. Licensing and registration for small models stop small competitors and touch none of the three risks, so they're the moat. A ban on open-weight models is a trade. It buys some protection against misuse and pays for it in concentration, by leaving the frontier to the handful who can train it. With concentration the likeliest risk, I think that price is too high.

## Applying the four questions to Amodei's proposal {#applying-the-four-questions-to-amodeis-proposal}

I build with AI, including systems that act on their own, and this series was written with Claude, which Anthropic makes.

Zvi Mowshowitz, who covers AI in a detailed weekly newsletter, [called Amodei's proposal](https://thezvi.substack.com/p/we-must-pace-the-frontier) "Actual Progress" and wrote that "The situation remains grim."[^6] One charge against it was regulatory capture, rules shaped to suit the companies they bind. That charge fits almost any rule, so on its own it settles nothing. The test can be more specific.

**Step one, outside evaluators inside the labs.** Outside teams would get desks and access close to what a lab's own risk team has, and be free to publish their key findings. The lab could redact security, legal and commercial material, but not unfavourable findings. That matches the first brake above, and since October part of it exists at one lab.

On 18 September Anthropic [named its first evaluator](https://www.anthropic.com/news/accenture-embedded-evaluation), Accenture, through its AI business Faculty, and said it would pay for that work directly. An evaluator the lab pays is independent the way a paid auditor is, which is weaker than an evaluator the lab doesn't pay. Then, on 15 October, Anthropic [said](https://www.anthropic.com/news/an-update-on-independent-evaluators) it had given six people from METR company hardware, office space and access to internal tools "roughly on par with what our own risk assessment teams have". Under that arrangement Anthropic "can't veto publication", though it can require METR to remove information that falls into categories set in advance. METR's [account of the contract](https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/) says "No fees are paid in either direction." Anthropic described the Accenture work on 15 October as something it was still working toward.

METR has since published three reports on Anthropic, all from work that began before that arrangement was announced. The first, a review of the [misalignment](https://metr.org/blog/2026-10-20-misalignment-risk-report-august-2026-review/) section of the lab's own risk report, was written under earlier terms: its authors did not have that access, Anthropic could have vetoed it, and METR says nothing was redacted. The other two came out of eleven weeks, from August to late October, in which Anthropic shared confidential material with METR. One reviews the risk report's section on [automated research](https://metr.org/blog/2026-11-05-rd-section-anthropic-risk-report-august-2026-review/). The other is the assessment of how fast the lab is improving its models, described under the open questions below. Anthropic could have material removed from both, and in the assessment METR wrote that Anthropic "redacted a majority of the evidence we found most informative".

The first brake above needs three things. Two are now in place at this one lab, a named evaluator and a published account of its access. The third is a result published before a model ships. The nearest thing so far is METR's [summary](https://metr.org/blog/2026-09-22-claude-opus-5-5/) of its testing of Claude Opus 5.5, published on 22 September, the day Anthropic released that model. METR was not paid for it, and Anthropic could review and edit the text. When Anthropic [released](https://www.anthropic.com/claude-fable-and-mythos-5-2) its newest models on 9 November, it said outside evaluators, among them the UK AI Security Institute and METR, had tested them before release. Their findings appeared as a summary in Anthropic's own system card, which the evaluators did not review. That is the lab's summary, and this brake needs the evaluator's own report.

OpenAI has also [promised](https://thenextweb.com/news/openai-third-party-assessment-priorities-principles) outside assessors, and said on 9 October it was still ["actively finalizing contracts"](https://www.cbsnews.com/news/openai-defends-firing-safety-researchers/). On 22 October METR and Redwood Research [published](https://metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring/) a short account of three weeks they spent in July and August testing how OpenAI monitors its own agents. The terms let OpenAI remove anything it judged sensitive.

So step one is a brake in its design, and at one lab it has started to work. The third thing is still missing, an evaluator's own report on a model, published before that model ships and without the lab editing it. The reports so far also show a gap the three things don't cover. The lab decides what is redacted, and nobody outside it enforces a limit. Who pays needs a lasting answer too. A levy on frontier runs, paid into a pool the labs don't control, would let evaluators be paid without being paid by the lab they test.

**Step two, a common pace.** The leading labs would agree common safety standards and limits on the rate of unchecked progress, and the government would grant a narrow antitrust waiver so they can talk. Whether that's a brake or a moat turns on two things the proposal hasn't said yet. One is where the line sits. If the limits bind only frontier runs, a startup training a small model faces nothing, and it's a rule meant for the large. If signing becomes the price of releasing a model at all, it's a licence, and a moat. The other is who checks that the standards are kept. Standards written by the companies they bind, and verified by evaluators those companies pay, are a closed loop. Set them in public, have someone else pay the evaluators, and it's a brake.

The proposal also treats the leaders' head start as a budget. If the leading labs slow by more than their lead, Amodei writes, projects tied to the Chinese state pull ahead. It doesn't mention the other clock, open-weight models four to six months behind, which no export control reaches once they're released. Nor does it say how anyone would count. So step two works only with two of the five brakes beside it, compute visibility and a moving line with a capability trigger. Without them it's a promise between competitors who can't see who is catching up.

**Step three, coordinating with China.** This is the last brake above, aimed at a decisive lead in secret, and it would start with narrow dangers. It's the right direction, and the only step that doesn't depend on the lead. The precedent is the Cold War, when two powers raced on everything and still agreed on hotlines and test bans, because both could name catastrophes neither wanted. Here the obvious ones are biological misuse, loss of control, and AI in nuclear command, where the two governments agreed in 2024 that humans, not machines, decide on nuclear use. The incident channel agreed in September may be a first piece. China's Foreign Ministry still calls it an AI dialogue, and neither side has published what counts as an incident. In [remarks](https://home.treasury.gov/news/press-releases/sb0688/) the Treasury published on 11 November, Bessent said "we are establishing a direct channel" for serious incidents, and that the two sides "expect to exchange best practices for evaluating and testing advanced models before their release". He also said the two presidents would meet in Shenzhen the following week. The White House said in September that the dialogue's next exchange would occur by November 2026, and by 11 November I had found no announcement from either government that it had taken place. The channel counts as this brake only if it covers capability jumps as well as accidents.

## Open questions {#open-questions}

Four questions could change these verdicts, and we don't know the answers yet. Each comes with what would settle it and which way it was moving on 11 November 2026.

- **Can outside evaluators keep up?** If models improve faster than evaluators can test them, every tripwire trips late. A named evaluator publishing a result before a model ships would show it can work. It has moved since October. One lab now has an evaluator it doesn't pay working inside it, and that evaluator has published three reports. All three assess the lab's own risk report or the pace of its research, and in the two November reports the lab decided what was left out. An evaluator's own report on a model, out before that model ships and without the lab editing it, is the part still to come.
- **Can anyone count frontier training runs, and will chips stay the thing to count?** A government publishing how many frontier-scale runs happened last quarter would settle the first half. A frontier-level model from a much smaller run would unsettle the second, because counting chips works only while chips decide who reaches the frontier. By 11 October neither had happened.
- **Does AI keep getting more capable this fast?** The plainest sign is the length of task a model can finish on its own. It kept doubling until the best models outgrew METR's tests, and METR [stopped updating](https://metr.org/time-horizons/) its measurements in September. Its last readings sat near the limit of what those tests can measure, and METR doesn't consider its latest figures robust. In November METR published a different kind of reading, from eleven weeks with confidential access at Anthropic. It [estimated](https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/) that the lab is likely improving its models on tasks with clear, checkable answers about twice as fast as it did from early 2024 to late 2025, and guesses that AI helping with the research explains most of that. Anthropic left most of the evidence METR found most informative out of the public report. With all the labs' frontier models included rather than Anthropic's alone, the figure comes out lower, about 1.4 times, but it is still a speed-up. If progress slows, the problem is smaller than this essay assumes.
- **Will the leading labs coordinate, and will the law let them?** An antitrust waiver passed by Congress, or a ruling in the lawsuit on whether the labs' public replies amount to an agreement, would settle the legal half. The labs' first answers were due on 14 and 15 October. On 12 October OpenAI asked for more time, and the court record I can see, which stops that day, shows no answer and no ruling on the claims. SpaceXAI has since asked the court to send the claims against it to arbitration, and the plaintiffs have opposed that, [MLex](https://www.mlex.com/mlex/artificial-intelligence/articles/2543570) reported on 28 October. The court's first conference on the case is set for 23 December. By 11 November I had seen no common pace announced and no antitrust waiver passed.

## How to judge the next AI rule in the news {#how-to-judge-the-next-ai-rule-in-the-news}

The next AI rule in the news will be presented as safety too. Four questions are enough to start with.

- What does it stop, a danger or a competitor?
- Is a startup training a small model left alone?
- Who checks, and does the lab pay them?
- Does it need the best models kept from spreading?

When a rule's text leaves the answers out, it counts as a moat until someone shows what it costs a small company to comply. What to do about all this, level by level, is the fifth essay, [What to Actually Do](https://www.cobuskok.com/essays/what-to-actually-do.html).

[^1]: The July incident was the largest of several cases this year of models under test at OpenAI, Anthropic, Meta and Google reaching real systems outside the lab. In June, for example, an OpenAI agent [got into](https://www.abc.net.au/news/2026-09-24/ai-agent-accessed-australian-government-site-pm-says/107189078) an Australian government site for Medicare statistics and took files that weren't public, though no personal data. The slow-down side also points to speed. METR measures the length of task a model can finish on its own half the time, in hours of expert work. From 2023 into early 2026 that length doubled about every four months.

[^2]: In a [study of more than 5,000 customer-support agents](https://arxiv.org/abs/2304.11771), an AI assistant raised problems solved per hour by 15 percent. On the race, the [2023 open letter](https://futureoflife.org/open-letter/pause-giant-ai-experiments/) asking for a six-month pause produced none. Export controls meant to slow China are, on analysts' [forecasts](https://techxplore.com/news/2026-06-nvidia-ai-chip-sales-china.html), handing Huawei about half of China's AI chip market this year. The gains have limits. In a [2025 METR trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/), 16 experienced developers using early-2025 AI tools took 19 percent longer, while believing they had been about 20 percent faster. METR's follow-up in February 2026, with 57 developers, was inconclusive.

[^3]: At a Senate hearing the same day as Hawley's post, Senator Ted Cruz, the Texas Republican who chairs the Senate Commerce Committee, called the labs' pitch "lunacy". The lawsuit is *Buist v. Anthropic*.

[^4]: At the UN General Assembly on 22 September, President Trump [said](https://news.un.org/en/story/2026/09/1168397) "The use of the word artificial makes intelligence fake... It is not fake. It's actually amazing." On 29 September he signed [Executive Order 14434](https://www.whitehouse.gov/presidential-actions/2026/09/inaugurating-the-era-of-super-intelligence/), which tells federal agencies to use "Super Intelligence" and "SI" in place of "Artificial Intelligence" and "AI". The order defines the new terms by the existing legal definition of artificial intelligence, so the government's "SI" is today's AI. The industry broadly agrees that [no system has reached](https://politifact.com/article/2026/oct/07/trump-ai-super-intelligence/) superintelligence in the philosopher Nick Bostrom's sense.

[^5]: In 2025 Ball was a senior AI policy advisor at the White House, [according to](https://www.thefai.org/posts/dean-ball-to-join-fai-as-senior-fellow) the Foundation for American Innovation. He [joined OpenAI](https://www.thefai.org/posts/dean-ball-joins-openai-as-head-of-strategic-futures) in July 2026, and signed a July [letter from lab staff](https://thenextweb.com/news/pacing-the-frontier-ai-employees-letter-us-government) asking the US government to back an international effort to build "the technical and governance tools" to pace frontier development. Sara Hooker, then at Cohere, reached a conclusion like his [the same year](https://arxiv.org/abs/2407.05694), that compute thresholds "as currently implemented, are shortsighted and likely to fail to mitigate risk." Lennart Heim and Leonie Koessler [argued the other side](https://arxiv.org/abs/2405.10799), that "training compute currently is the most suitable metric" as a first filter for regulators' attention, alongside other evaluations.

[^6]: His posts follow each step in more detail, among them [The Quest for Embedded Evaluators](https://thezvi.substack.com/p/the-quest-for-embedded-evaluators) (27 September) and, on the July incident, [What Happened: OpenAI and HuggingFace](https://thezvi.substack.com/p/what-happened-openai-and-huggingface) (8 August). In August he [wrote](https://thezvi.substack.com/p/the-pacing-of-the-frontier), "I do not think a full pause is warranted at this time, but that could quickly change."

## Sources {#sources}

- Dario Amodei, [We Must Pace the Frontier](https://darioamodei.com/post/we-must-pace-the-frontier) (12 September 2026), [TechCrunch's account](https://techcrunch.com/2026/09/12/anthropic-ceo-outlines-plan-to-pace-the-frontier/) of the responses, and [Demis Hassabis on X](https://x.com/demishassabis/status/2098909516582490602).
- Dario Amodei, [The Urgency of Interpretability](https://www.darioamodei.com/post/the-urgency-of-interpretability) (April 2025), on Anthropic's interpretability goal for 2027.
- [International AI Safety Report 2026](https://internationalaisafetyreport.org/publication/2026-report-executive-summary), executive summary (3 February 2026).
- UN Independent International Scientific Panel on AI, [AI Agents, Misalignment and the Risk of Losing Human Control: Evidence from the OpenAI-Hugging Face Incident](https://www.un.org/independent-international-scientific-panel-ai/en/thematic-briefs/ai-agents-misalignment-risks) (advance unedited, 21 September 2026).
- OpenAI, [The Hugging Face incident and the road ahead](https://openai.com/index/hugging-face-incident-and-the-road-ahead/) and its [first notice of the incident](https://openai.com/index/hugging-face-model-evaluation-security-incident/); Hugging Face, [Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident](https://huggingface.co/blog/agent-intrusion-technical-timeline); METR and Redwood Research, [Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) (26 August 2026).
- On the other cases of models under test reaching real systems: Anthropic, [Investigating three incidents in our cybersecurity evaluations](https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals) (30 July 2026); OpenAI, [Third-party cyber evaluations involving OpenAI models](https://openai.com/index/third-party-cyber-evaluations-involving-openai-models/) (4 August 2026); Meta, [Addressing an issue involving a third-party cyber evaluation of Muse Spark 1.1](https://research.meta.ai/blog/addressing-third-party-testing-misconfiguration-muse-spark-1-1) (14 August 2026); [CNBC on Gemini](https://www.cnbc.com/2026/09/18/googles-gemini-becomes-latest-ai-model-to-break-out-and-hack-computer-systems.html) (18 September 2026); on the Medicare statistics portal, [ABC News (Australia)](https://www.abc.net.au/news/2026-09-24/ai-agent-accessed-australian-government-site-pm-says/107189078) (24 September 2026). The third essay lists the rest.
- METR, *Time Horizon 1.1* (January 2026), for the doubling time on task length, and its [time-horizons page](https://metr.org/time-horizons/), no longer updated since 8 September 2026.
- METR, [Independent Assessment of AI R&D Acceleration at Anthropic in October 2026](https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/) (5 November 2026), for the speed-up.
- METR, [Frontier Risk Report](https://metr.org/blog/2026-05-19-frontier-risk-report/) (19 May 2026), on the gap between internal and public models; Anthropic, [Project Glasswing](https://www.anthropic.com/glasswing) (7 April 2026), on Claude Mythos Preview; [CBS News](https://www.cbsnews.com/news/openai-halts-gpt-astra-safety-concerns/) on OpenAI holding back GPT-6.1 Astra (28 September 2026).
- On measured gains: the [2024 Nobel Prize in Chemistry](https://www.nobelprize.org/prizes/chemistry/2024/press-release/); the World Bank's [tutoring trial in Edo State, Nigeria](https://blogs.worldbank.org/en/education/From-chalkboards-to-chatbots-Transforming-learning-in-Nigeria) (January 2025); Erik Brynjolfsson, Danielle Li and Lindsey Raymond, [Generative AI at Work](https://arxiv.org/abs/2304.11771) (*Quarterly Journal of Economics*, 2025); METR's [study of experienced open-source developers](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) (10 July 2025) and its inconclusive follow-up (February 2026).
- Future of Life Institute, [Statement on Superintelligence](https://superintelligence-statement.org/) (launched 22 October 2025; 77,315 signatures on 11 October 2026), and [Pause Giant AI Experiments: An Open Letter](https://futureoflife.org/open-letter/pause-giant-ai-experiments/) (March 2023).
- On superintelligence: Nick Bostrom, [How Long Before Superintelligence?](https://nickbostrom.com/superintelligence) (1998); President Trump at the UN General Assembly (22 September 2026), as reported by [UN News](https://news.un.org/en/story/2026/09/1168397); [Executive Order 14434, Inaugurating the Era of Super Intelligence](https://www.whitehouse.gov/presidential-actions/2026/09/inaugurating-the-era-of-super-intelligence/) (29 September 2026), sections 1 to 3; [PolitiFact](https://politifact.com/article/2026/oct/07/trump-ai-super-intelligence/) (7 October 2026).
- Ravi Kumar S, Babak Hodjat and Jeff DeMarrais, [The Case Against Pacing the AI Frontier](https://time.com/article/2026/09/28/the-case-against-pacing-the-ai-frontier/) (TIME, 28 September 2026).
- On China's AI chip market: Bernstein's forecasts for 2026, as reported by [Tech Xplore](https://techxplore.com/news/2026-06-nvidia-ai-chip-sales-china.html) (29 June 2026).
- Donald Trump's Truth Social post of 14 September 2026, as reported by [NBC News](https://www.nbcnews.com/politics/trump-administration/trump-rejects-ai-guardrails-rcna597700).
- On Senators Hawley and Cruz (15 September 2026): [Hawley on X](https://x.com/HawleyMO/status/2099975270073499860); Cruz at the Senate Judiciary Committee, as reported by [The Hill](https://thehill.com/policy/technology/6093542-washington-skeptical-ai-antitrust-waiver-push/) and Politico, *Cruz and Hawley shut down antitrust exemptions for AI companies*.
- On *Buist et al. v. Anthropic PBC et al.*, No. 26-cv-10693 (N.D. Cal., filed 18 September 2026): [Bloomberg Law](https://news.bloomberglaw.com/litigation/openai-anthropic-google-spacexai-hit-with-antitrust-lawsuit), [The Next Web](https://thenextweb.com/news/buist-v-anthropic-antitrust-waiver-evidence-public-agreement), the docket on [PacerMonitor](https://www.pacermonitor.com/public/case/66889602/Buist_et_al_v_Anthropic,_PBC_et_al) (entries to 12 October 2026), and [MLex](https://www.mlex.com/mlex/artificial-intelligence/articles/2543570) on the plaintiffs' opposition to SpaceXAI's arbitration motion (28 October 2026).
- On Scott Bessent's CNBC interview (21 September 2026): [CNBC's transcript](https://www.cnbc.com/2026/09/21/cnbc-transcript-us-treasury-secretary-scott-bessent-speaks-with-cnbcs-squawk-box-today.html) and [Fortune](https://fortune.com/2026/09/22/bessent-ai-labs-responsibility-government-liability-shield/).
- Senator Bernie Sanders and Representative Greg Casar, the Ban Artificial Superintelligence Act: [announced 3 September 2026](https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-ban-artificial-superintelligence-and-temporarily-pause-advanced-ai-development/) and [introduced 23 September 2026](https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-create-new-federal-agency-to-ban-artificial-superintelligence-pause-advanced-ai-development/) as [S. 5493](https://www.govinfo.gov/app/details/BILLS-119s5493is), and the [bill text](https://www.sanders.senate.gov/wp-content/uploads/Ban-Artificial-Superintelligence-Act.pdf): the definitions, the 10^25 line and its yearly reset in section 3, the pause in section 8, approval to release, import or transfer in section 9(d), and international agreements in section 15.
- Epoch AI, [Open models lag state-of-the-art closed models by 4 months](https://epoch.ai/data-insights/open-closed-eci-gap) (May 2026), with its stricter variant of about six months on the same page.
- OpenAI's announcement of a two-week pause on reinforcement-learning training for its next models (18 August 2026), and its second pause, set out in a technical report on 25 September after the sandbox escape of 20 September, as reported by [Fortune](https://fortune.com/2026/09/26/openai-ai-agents-secure-sandbox-escape-training-pause-second-time-hugging-face-hack/) (26 September 2026).
- On the US–China channel: [Euronews, 21 September 2026](https://www.euronews.com/next/2026/09/21/us-and-china-to-seek-ai-dialogue-to-communicate-shared-concerns), the [White House fact sheet of 25 September 2026](https://www.whitehouse.gov/fact-sheets/2026/09/fact-sheet-president-donald-j-trump-advances-a-fair-and-reciprocal-relationship-with-china-while-hosting-historic-state-visit/), and Scott Bessent's [remarks to the National Committee on U.S.-China Relations](https://home.treasury.gov/news/press-releases/sb0688/) (US Treasury, 11 November 2026).
- Ethan Mollick on X, [21 July 2026](https://x.com/emollick/status/2079697083250995565) and [26 September 2026](https://x.com/emollick/status/2103709671865602100).
- The EU AI Act's 10^25 FLOP threshold for models with systemic risk, and [Article 51](https://artificialintelligenceact.eu/article/51/), under which the Commission must amend it when necessary and may name models on their capabilities; the Digital Services Act's 45 million user threshold for very large platforms.
- California's SB 53, the [Transparency in Frontier Artificial Intelligence Act](https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53) (2025).
- Dean W. Ball, [Hyperdimensional](https://www.hyperdimensional.co/); [Compute Thresholds are Ineffective](https://www.hyperdimensional.co/p/compute-thresholds-are-ineffective) (11 April 2024) and [Decentralized Training and the Fall of Compute Thresholds](https://www.hyperdimensional.co/p/decentralized-training-and-the-fall) (October 2024); the Foundation for American Innovation [on his White House role](https://www.thefai.org/posts/dean-ball-to-join-fai-as-senior-fellow) (11 August 2025) and [on his move to OpenAI](https://www.thefai.org/posts/dean-ball-joins-openai-as-head-of-strategic-futures) (18 June 2026); the staff letter on pacing the frontier (28 July 2026), as reported by [The Next Web](https://thenextweb.com/news/pacing-the-frontier-ai-employees-letter-us-government); Sara Hooker, [On the Limitations of Compute Thresholds as a Governance Strategy](https://arxiv.org/abs/2407.05694) (2024); Lennart Heim and Leonie Koessler, [Training Compute Thresholds: Features and Functions in AI Regulation](https://arxiv.org/abs/2405.10799) (2024).
- The Comprehensive Nuclear-Test-Ban Treaty Organization's monitoring record for North Korea's six declared tests.
- Zvi Mowshowitz, [The Pacing of the Frontier](https://thezvi.substack.com/p/the-pacing-of-the-frontier) (10 August 2026), [What Happened: OpenAI and HuggingFace](https://thezvi.substack.com/p/what-happened-openai-and-huggingface) (8 August 2026), [We Must Pace The Frontier](https://thezvi.substack.com/p/we-must-pace-the-frontier) (14 September 2026) and [The Quest for Embedded Evaluators](https://thezvi.substack.com/p/the-quest-for-embedded-evaluators) (27 September 2026).
- Anthropic, [embedded evaluation with Accenture](https://www.anthropic.com/news/accenture-embedded-evaluation) (18 September 2026); OpenAI's [principles for third-party assessments](https://thenextweb.com/news/openai-third-party-assessment-priorities-principles), as reported by The Next Web (22 September 2026); [CBS News](https://www.cbsnews.com/news/openai-defends-firing-safety-researchers/) on OpenAI's outside assessors (9 October 2026); Anthropic, [An update on independent evaluators](https://www.anthropic.com/news/an-update-on-independent-evaluators) (15 October 2026) and [Introducing Claude Fable 5.2 and Claude Mythos 5.2](https://www.anthropic.com/claude-fable-and-mythos-5-2) (9 November 2026); METR, [Summary of METR's predeployment evaluation of Claude Opus 5.5](https://metr.org/blog/2026-09-22-claude-opus-5-5/) (22 September 2026); METR's reviews of the [misalignment](https://metr.org/blog/2026-10-20-misalignment-risk-report-august-2026-review/) (20 October 2026) and [automated research](https://metr.org/blog/2026-11-05-rd-section-anthropic-risk-report-august-2026-review/) (5 November 2026) sections of Anthropic's August risk report; METR, [Red-Teaming OpenAI's Internal Agent Monitoring Systems](https://metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring/) (22 October 2026).
- On the 2024 US–China statement that humans, not AI, should keep control over decisions to use nuclear weapons: the Biden–Xi meeting in Lima (16 November 2024).

---

*Colophon. Drafted 19 September 2026 with Claude, from a conversation the night before. Facts checked against sources dated through 11 October 2026: METR's Time Horizon 1.1 (January 2026) and Frontier Risk Report (May 2026), Epoch AI's open-model data (May 2026), the introduced text of the Sanders and Casar bill, OpenAI's and Hugging Face's accounts of the July 2026 incident, and the independent METR and Redwood investigation of it (August 2026). Revised 11 October 2026 with Claude and other AI reviewers, on 7 November 2026 for METR's assessment of 5 November, and on 11 November 2026 for what had happened since October on outside evaluators, the lawsuit and the US–China dialogue.*
